API Token Authorization and Refreshing the Token in Tadabase

 

Hello Tadabase Users,

In this episode of Build It With Tim, we will explore the essential process of dealing with API token authorization and refreshing tokens using Tadabase. This functionality is crucial when working with external APIs that require authentication via tokens, and these tokens need periodic refreshing to maintain access. We will illustrate this concept using Octoparse as an example, a powerful tool for web scraping and data extraction.

Goal: The primary goal of this tutorial is to guide you through the process of setting up a custom pipe in Tadabase to handle token authorization and token refreshing for seamless integration with APIs like Octoparse. We'll cover the following key aspects:

1. Obtaining an initial token.
2. Refreshing the token to ensure it remains valid.
3. Automating the process to maintain a continuous flow of tokens.

Getting Started: First, you'll need to understand your API's specific requirements for token authorization. In this example, we use Octoparse, but the principles can be applied to other APIs.

Setting Up the Custom Pipe:

1. Access the builder in Tadabase.
2. Create a custom pipe (e.g., "Octoparse").
3. Configure three API calls: a. Obtain the initial token. b. Refresh the token. c. Execute a task using the obtained token.

Obtaining the Initial Token: For Octoparse, we send a POST request to a specific endpoint with a content type header of "application/json." The request includes the username, password, and grant type. The response will contain an access token and a refresh token. Save both values.

Refreshing the Token: To refresh the token, send another POST request to the same endpoint with the refresh token as a parameter. Save the new access and refresh tokens from the response.

Executing a Task: Now that you have a valid token, you can use it to perform actions with the API. Set up the request with necessary parameters, including the access token in the header. Test the request to ensure it works correctly.

Automating Token Refresh: To automate the token refreshing process, follow these steps:

1. Create a table to store relevant data, including task IDs, access tokens, and refresh tokens.
2. Set up a table rule to trigger token retrieval whenever a record is created or edited.
3. Schedule a task to refresh the token daily at a specific time, updating the record with the new tokens.

By implementing these steps, you ensure that your tokens remain valid, and you can seamlessly interact with the API without worrying about expirations.

Please note that API integration can vary widely, and this tutorial is specific to Octoparse. If you have questions or need guidance for a different API, feel free to reach out to our support team.

Thank you for tuning in to this episode of Build It With Tim. We hope this tutorial helps you effectively handle token authorization and refreshing, making your API integrations smoother and more reliable. Stay tuned for more valuable Tadabase insights in our future episodes!

Features Discussed:

1. Custom Pipe (Time: 2:15)
   Learn how to create a custom pipe that integrates with an external API. For example, learn how to set up API calls to obtain, refresh, and use tokens, which is critical for managing secure connections and task executions with an API like Octoparse.
2. Custom Pipe (Time: 2:30)
   Learn how to create a custom pipe for obtaining an API token. For example, you’ll see how to send a POST request with JSON data, including username and password, to retrieve an access token and a refresh token from an external API.
3. Custom Pipe (Time: 4:50)
   Learn how to set up a custom pipe for refreshing an API token. For example, the video demonstrates sending another POST request using the refresh token to obtain a new access token.
4. Custom Pipe (Time: 6:30)
   Learn how to initiate a task using the retrieved tokens. For example, discover how to send a POST request with task details to start a specific task in the external system using the obtained access token.
5. Table Rules (Time: 9:45)
   Learn how to store access and refresh tokens securely by utilizing a long text field for the access token due to its size, and a regular text field for the refresh token. For example, see how to set up a table rule to automatically retrieve and store these tokens whenever a record is created or edited.
6. Scheduled Tasks (Time: 10:50)
   Learn how to automate the refresh of access tokens using a scheduled task in Tadabase. This process ensures your tokens remain current by running daily tasks at a specified time. For example, set it to midnight to refresh tokens, update records, and trigger API requests consistently without manual intervention.

 

---

Transcript Summary

Introduction

Hey everybody, this is Build It with Tadabase. On today's episode, we're going to be dealing with some external APIs using Tadabase Pipes. Specifically, we're going to talk about APIs that require a token as a form of authorization, and they require that token to be refreshed after a certain amount of time.

Example Overview: Octoparse API

For this example, we're going to be dealing with Octoparse, which is a great tool for scraping websites and extracting data. We're going to build a custom pipe to support this, and I'll show you the technique for how to get a token in the first place and then how to send a refresh request to get a new token.

Setting Up the Task in Octoparse

In Octoparse, I've built a task that scrapes a website to get announcement posts. We're going to extract that data on a schedule, say once a day. In this application, I've created a standalone table called "Octoparse Web Scraping Tasks," where I've stored the task ID that comes from Octoparse.

Understanding the API Authentication Process

Before triggering the task via API, we need to get authenticated by requesting a token and ensuring that the token stays alive, meaning it doesn't expire within the set expiration time.

Setting Up a Custom Pipe

We'll set up a custom pipe in Tadabase to handle this process.

Step 1: Create the Custom Pipe

1. Add the Custom Pipe:

• Click "Add Custom Pipe" and name it "Octoparse."
• Add three API calls: "Get Token," "Refresh Token," and "Start Task."

1. Configure the 'Get Token' Call:

• Set up a POST request to the Octoparse API endpoint.
• The request includes a Content-Type header of application/json and a raw JSON request body containing the username, password, and grant type.
• Test the call to receive an access token, expiration time, token type, and refresh token.
• Save the access token and refresh token by mapping these values in the response section.

Step 2: Set Up the 'Refresh Token' Call

1. Configure the 'Refresh Token' Call:

• Set up another POST request to the same endpoint, but this time with a request body that includes the refresh token.
• Save the refresh token dynamically as a parameter and pass it in the request.
• Test the call to receive a new access token and refresh token, and save these values.

Step 3: Set Up the 'Start Task' Call

1. Configure the 'Start Task' Call:

• Set up a POST request to trigger the task in Octoparse, passing the task ID in the request body.
• The access token is passed in the header as authorization.
• Test the call by manually entering the task ID and access token to ensure the request triggers the task.

Automating the Token Management

Step 1: Store Tokens in a Data Table

1. Set Up Data Fields:

• Add fields in the Octoparse Web Scraping Tasks table to store the access token and refresh token.
• Use a long text field for the access token and a regular text field for the refresh token.

1. Create a Table Rule:

• Set up a table rule to get a token whenever a record is created or edited.
• Trigger the "Get Token" pipe, and map the access token and refresh token to their respective fields in the table.

Step 2: Automate Token Refresh

1. Set Up a Scheduled Task:

• Create a scheduled task to run every 24 hours at midnight.
• Trigger the "Refresh Token" pipe, passing the stored refresh token and saving the new access token and refresh token back to the table.

Conclusion

Now, every time a record is created, the application will automatically generate and store the access token and refresh token. The scheduled task ensures the token is refreshed daily, so it never expires, and you always have a valid token ready to use.

I hope this helps you understand how to deal with token authorization, specifically tokens that expire, and how to refresh them for continued authorization in subsequent API requests. Every external API will be different, so if you have any questions, feel free to reach out to our support team. We'd be happy to help. Thanks so much for watching this week's episode of Build It with Tadabase. We'll see you next time. Take care!