Pipes are Tadabase's powerful feature for connecting to external APIs and services to retrieve and process data dynamically. Think of pipes as bridges that allow data to flow from external sources into your Tadabase application in real-time. Unlike static data stored in your tables, pipes fetch fresh data on demand from external sources. This enables your applications to display current weather, validate addresses, check inventory from suppliers, retrieve shipping rates, perform currency conversions, and connect to virtually any web service that provides an API.

Pipes solve several important challenges in application development:

Real-Time Data Access

• Display current information without storing it locally
• Ensure data is always up-to-date
• Reduce data storage requirements
• Avoid manual data synchronization

Extend Functionality

• Leverage specialized external services (payment processing, geolocation, etc.)
• Access data from systems you don't control
• Integrate with proprietary databases and APIs
• Add capabilities without custom development

Dynamic Form Behavior

• Populate dropdowns based on user selections
• Validate user input against external databases
• Calculate values based on external data (shipping costs, tax rates)
• Provide real-time feedback as users fill forms

Data Transformation

• Use Tadabase functions to manipulate text, dates, and numbers
• Format data for display or processing
• Perform calculations using pipe results
• Chain multiple operations together

To understand pipes better, let's look at practical examples:

Example 1: Address Validation

Scenario: You have a customer form where users enter their address. You want to validate the zip code and automatically populate the city and state. How pipes help:

• User enters zip code
• Pipe calls postal service API with the zip code
• API returns city and state information
• Form fields automatically populate with correct values
• Invalid zip codes show error messages

Benefits: Ensures accurate addresses, reduces user errors, improves data quality.

Example 2: Shipping Rate Calculator

Scenario: An order form needs to calculate shipping costs based on destination and weight. How pipes help:

• User enters shipping address and package weight
• Pipe calls shipping carrier API (UPS, FedEx, USPS)
• API returns available shipping options and rates
• Dropdown populates with shipping choices and prices
• User selects preferred shipping method

Benefits: Real-time pricing, multiple carrier options, accurate cost calculation.

Example 3: Currency Conversion

Scenario: Your application handles international transactions and needs current exchange rates. How pipes help:

• User enters amount in their local currency
• Pipe calls currency exchange API
• Current exchange rate retrieved
• Converted amount displayed in real-time
• Rates update automatically throughout the day

Benefits: Accurate conversions, current rates, multi-currency support.

Example 4: Inventory Lookup

Scenario: Sales team needs to check product availability from supplier's system. How pipes help:

• User searches for product by SKU
• Pipe queries supplier's API
• Current stock levels retrieved
• Availability and lead times displayed
• Pricing information shown

Benefits: Real-time inventory, accurate availability, no duplicate data entry.

Understanding how pipes work under the hood helps you use them effectively:

Request Flow

1. Trigger: User action or page load initiates the pipe
2. Input: Tadabase sends data to the external API (parameters, authentication)
3. API Call: External service processes the request
4. Response: API returns data (usually in JSON format)
5. Processing: Tadabase parses the response and extracts needed values
6. Display: Data appears in your application (form field, component, etc.)

Key Components

API Endpoint: The URL where the external service receives requests

• Example: https://api.weather.com/v1/current

HTTP Method: The type of request (GET, POST, PUT, DELETE)

• GET: Retrieve data
• POST: Send data
• PUT/PATCH: Update data
• DELETE: Remove data

Headers: Metadata sent with the request

• Authentication tokens
• Content type (JSON, XML, etc.)
• API keys
• Custom headers required by the service

Parameters: Data sent to the API

• Query parameters (in the URL)
• Body parameters (in the request body)
• Path parameters (part of the URL)

Response: Data returned from the API

• JSON objects and arrays
• Status codes (200 = success, 404 = not found, etc.)
• Error messages
• Nested data structures

Tadabase offers several types of pipes, each designed for specific purposes:

1. External API Pipes

Connect to third-party APIs and web services. Use cases:

• Payment processing (Stripe, PayPal)
• Geolocation services (Google Maps, Mapbox)
• Communication services (Twilio, SendGrid)
• Data validation (address verification, email validation)
• Business data (company information, credit checks)

Configuration:

• API endpoint URL
• Authentication method (API key, OAuth, Basic Auth)
• Request parameters
• Response mapping

2. Tadabase Function Pipes

Use built-in Tadabase functions to manipulate data. Text functions:

• upper(text) - Convert to uppercase
• lower(text) - Convert to lowercase
• trim(text) - Remove leading/trailing spaces
• substring(text, start, length) - Extract portion of text
• concat(text1, text2, ...) - Combine text values
• replace(text, find, replace) - Find and replace text

Date functions:

• now() - Current date and time
• today() - Current date
• dateAdd(date, days) - Add days to a date
• dateDiff(date1, date2) - Calculate difference between dates
• formatDate(date, format) - Format date display

Number functions:

• round(number, decimals) - Round to specified decimals
• abs(number) - Absolute value
• min(num1, num2, ...) - Minimum value
• max(num1, num2, ...) - Maximum value
• sum(num1, num2, ...) - Add numbers

3. Custom API Pipes

Connect to your own APIs or internal systems. Use cases:

• Internal databases and systems
• Custom business logic
• Proprietary data sources
• Legacy system integration

Requirements:

• API must be accessible via HTTP/HTTPS
• Should return JSON or XML responses
• Proper authentication and security
• Documented endpoints and parameters

Creating a pipe involves several configuration steps. Let's walk through each element:

Basic Settings

Pipe Name:

• Descriptive name for identification
• Example: "Validate Zip Code" or "Get Shipping Rates"
• Use clear naming conventions for easy management

Description:

• Document the pipe's purpose
• Note any special requirements or limitations
• Include API documentation links

API Endpoint Configuration

URL:

• Full API endpoint URL
• Can include dynamic parameters using {field_name} syntax
• Example: https://api.service.com/lookup?zip={zip_code}

HTTP Method:

• GET - Retrieve data (most common for pipes)
• POST - Send data to create or process
• PUT - Update existing data
• DELETE - Remove data

Authentication

API Key:

• Most common authentication method
• Can be sent in headers or as query parameter
• Example header: Authorization: Bearer YOUR_API_KEY

Basic Authentication:

• Username and password encoded in header
• Format: Authorization: Basic base64(username:password)

OAuth:

• Token-based authentication
• Requires authorization flow
• More complex but more secure

No Authentication:

• Public APIs that don't require authentication
• Use cautiously due to rate limits

Headers

Common headers you might configure:

• Content-Type: application/json - Specify JSON data format
• Accept: application/json - Request JSON response
• Authorization: Bearer TOKEN - API authentication
• X-API-Key: YOUR_KEY - Alternative API key method
• Custom headers required by specific APIs

Parameters

Query Parameters:

• Appear in URL after "?"
• Example: ?zip=12345&country=US
• Use for GET requests

Body Parameters:

• Sent in request body (POST, PUT)
• Usually JSON format
• Example: {"zip": "12345", "country": "US"}

Dynamic Values:

• Reference form fields: {field_slug}
• Reference user data: {user.email}
• Reference record data: {record.field_name}

Response Mapping

JSON Path:

• Specify which part of the response to use
• Example: data.city extracts city from nested JSON
• Use dot notation for nested objects
• Use brackets for arrays: results[0].name

Default Values:

• Value to use if pipe fails or returns empty
• Ensures application continues working
• Example: "Unknown" for missing city names

Error Handling:

• What to display if API call fails
• User-friendly error messages
• Fallback behaviors

Pipes can be triggered in different ways depending on your needs:

On Page Load

• Pipe executes when page opens
• Useful for displaying initial data
• Example: Show current weather on dashboard
• Can slow page load if API is slow

On Field Change

• Pipe executes when specific field value changes
• Ideal for dynamic form behavior
• Example: Update city when zip code changes
• Provides real-time feedback to users

On Button Click

• User explicitly triggers the pipe
• Gives user control over API calls
• Example: "Validate Address" button
• Reduces unnecessary API calls

On Form Submit

• Pipe executes before form submission
• Used for final validation or data enrichment
• Example: Verify credit card before processing payment
• Can prevent form submission if validation fails

Different industries benefit from pipes in unique ways:

E-Commerce

• Calculate shipping rates from multiple carriers
• Verify credit card information
• Check real-time inventory from suppliers
• Validate addresses before shipping
• Currency conversion for international sales
• Tax calculation based on location

Real Estate

• Property valuation from Zillow/Trulia
• Mortgage rate calculations
• Address standardization
• Property tax lookup
• School district information
• Crime statistics by area

Healthcare

• Insurance verification
• Drug interaction checking
• Medical code lookup (ICD-10, CPT)
• Patient demographic verification
• Appointment availability checking
• Lab result integration

Logistics

• Real-time tracking information
• Route optimization
• Fuel price data
• Traffic and weather conditions
• Distance calculations
• Delivery time estimates

Financial Services

• Credit score checks
• Stock quotes and market data
• Currency exchange rates
• Interest rate calculations
• Identity verification
• Fraud detection scoring

Tadabase provides some pre-built pipes and also allows you to create custom ones:

Built-in Pipes

Advantages:

• Pre-configured and tested
• No setup required
• Maintained by Tadabase
• Reliable and performant
• Documentation included

Tadabase Function Pipes:

• Text manipulation (upper, lower, trim, etc.)
• Date operations (format, add, subtract, etc.)
• Number functions (round, format, calculate, etc.)
• Always available, no API keys needed

Third-Party Pipes

Characteristics:

• Connect to external services
• Require API keys/authentication
• Subject to external service limitations
• May have usage costs
• You configure and maintain them

Popular third-party APIs:

• Google Maps: Geocoding, directions, places
• Stripe: Payment processing
• Twilio: SMS and phone services
• SendGrid: Email delivery
• AWS: Various cloud services
• OpenAI: AI and language processing

Choosing Between Them

Use built-in pipes when:

• Simple data transformation needed
• No external dependencies desired
• Guaranteed uptime is critical
• No budget for API costs

Use third-party pipes when:

• Specialized functionality required
• Real-time external data needed
• Integration with specific services necessary
• Worth the cost for the functionality

Pipes call external services, which introduces performance considerations:

Latency

• Network delay: Time for request to reach API and return
• Processing time: How long API takes to process request
• Total impact: Can range from 100ms to several seconds
• User experience: Loading indicators help manage expectations

Rate Limits

• API restrictions: Most APIs limit requests per minute/hour/day
• Exceeded limits: API may block requests or charge extra
• Mitigation: Cache results, batch operations, upgrade API plan
• Monitoring: Track API usage to avoid limits

Error Handling

• API downtime: External service may be unavailable
• Invalid responses: API may return unexpected data
• Timeout: API may not respond within timeout period
• User impact: Provide fallbacks and clear error messages

Best Practices for Performance

• Cache when possible: Store frequently accessed data temporarily
• Lazy loading: Load data only when needed
• Minimize calls: Batch requests instead of multiple individual calls
• Async processing: Don't block user interface waiting for pipes
• Timeout settings: Set appropriate timeouts to prevent hanging
• Loading indicators: Show progress during pipe execution

When working with pipes, security is paramount:

API Key Protection

• Never expose keys: Don't include API keys in client-side code
• Tadabase storage: Store keys securely in pipe configuration
• Key rotation: Regularly update API keys
• Access control: Limit who can view/edit pipes

Data Validation

• Sanitize inputs: Validate data before sending to APIs
• Validate responses: Don't trust external data blindly
• Type checking: Ensure data types match expectations
• Length limits: Prevent injection attacks with size limits

HTTPS Only

• Encrypted connections: Always use HTTPS, never HTTP
• Certificate validation: Verify SSL certificates
• Man-in-the-middle protection: Prevent data interception

Error Message Security

• User messages: Generic, non-revealing error messages
• Logs: Detailed errors in logs for administrators only
• Information disclosure: Don't reveal system details in errors

Thorough testing ensures pipes work correctly:

Test Scenarios

• Happy path: Normal operation with valid data
• Invalid input: How pipe handles bad data
• API errors: What happens when API returns error
• Timeout: Behavior when API doesn't respond
• Empty response: Handling when API returns no data
• Malformed response: Dealing with unexpected data format

Testing Tools

• Tadabase Pipe Tester: Built-in tool for testing pipe configuration
• Postman: Test API endpoints before configuring in Tadabase
• Browser DevTools: Inspect network requests and responses
• JSON Validator: Verify JSON structure and path expressions

Debugging Tips

• Check logs: Review pipe execution logs for errors
• Verify endpoint: Ensure API URL is correct
• Test authentication: Confirm API keys and tokens work
• Validate parameters: Check that all required parameters are provided
• Inspect response: Look at actual API response structure
• Path expressions: Verify JSON path matches response structure

Understanding pipe limitations helps you design better solutions:

Technical Limitations

• Timeout: Maximum execution time (typically 30 seconds)
• Response size: Large responses may be truncated
• Complexity: Very complex API configurations may not be supported
• Authentication: Some OAuth flows may not work

Practical Limitations

• API costs: External API calls may have per-request charges
• Rate limits: APIs limit number of requests
• Availability: Dependent on external service uptime
• Data freshness: Real-time doesn't mean instant (slight delays exist)

Workarounds

• Webhooks: Use for real-time data synchronization
• Scheduled imports: For bulk data that doesn't need to be real-time
• Caching: Store frequently accessed data locally
• Multiple APIs: Use backup services for redundancy

Let's create a simple pipe to understand the concepts:

Exercise: Current Date Function Pipe

Goal: Create a pipe that automatically populates a date field with the current date. Steps:

1. Navigate to Builder → Pipes
2. Click Add New Pipe
3. Name: "Current Date"
4. Type: Tadabase Function
5. Function: today()
6. Save the pipe
7. Add to a form:
   • Open form builder
   • Add date field
   • In field settings, select your "Current Date" pipe as default value
8. Test by opening the form—date should auto-populate

What you learned:

• How to create a pipe
• Using Tadabase functions
• Connecting pipes to form fields
• Testing pipe execution

These patterns solve common integration challenges:

Pattern 1: Dependent Dropdowns

Scenario: Country selection determines available states/provinces.

• First dropdown: Countries (static list)
• Second dropdown: States (populated by pipe based on country selection)
• Pipe triggers when country changes
• API returns states for selected country

Pattern 2: Data Enrichment

Scenario: User enters email, system looks up company information.

• Email field with validation
• Pipe extracts domain from email
• API looks up company by domain
• Company name, size, industry auto-populate

Pattern 3: Real-Time Calculation

Scenario: Calculate product price with current exchange rate.

• User selects currency
• Pipe fetches current exchange rate
• Price automatically converts and displays
• Updates when currency selection changes

Pattern 4: Validation

Scenario: Verify business identification number is valid.

• User enters business ID
• Pipe queries government registry
• Returns valid/invalid status
• Form prevents submission if invalid

Now that you understand what pipes are and how they work, you're ready to implement them in your applications. In the next article, you'll learn how to use pipes in components—specifically how to:

• Load external data into form fields
• Create dynamic form values based on user input
• Validate forms using pipes
• Use Tadabase text, date, and number functions
• Chain multiple pipes together
• Handle errors and provide user feedback

The combination of understanding pipe architecture (this article) and practical implementation (next article) will give you complete mastery of this powerful feature.

Remember these essential pipe concepts:

• Purpose: Pipes connect external APIs to retrieve and process data dynamically
• Types: External APIs, Tadabase functions, and custom APIs
• Architecture: Request → API call → Response → Processing → Display
• Configuration: Endpoint, method, authentication, parameters, response mapping
• Execution: Can trigger on page load, field change, button click, or form submit
• Performance: Consider latency, rate limits, and error handling
• Security: Protect API keys, validate data, use HTTPS
• Testing: Thoroughly test all scenarios before production deployment