> ## Documentation Index > Fetch the complete documentation index at: https://microstrate-1133-notifications-prefs.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Data Transformation Utilities > Transform, format, and manipulate data in your flows # Data Transformation Utilities Utility functions for common data operations like encoding, merging, grouping, and formatting. These functions help you transform data between steps in your flows. **Quick Data Transformation**: These utilities handle common data manipulation tasks without needing custom code. Use them for encoding, merging objects, formatting templates, and more. *** ## Function List Encode to base64 Decode from base64 Convert JSON to XML Convert XML to JSON Dynamic templates JSON path data mapping Retrieve secret keys Invoke other functions Upload files via SFTP *** ## base64-encode Encode text or objects to base64 format. Automatically handles both string and object inputs, converting objects to JSON before encoding. ### Parameters Takes a single input value directly (not wrapped in a params object): The data to encode. Can be: * **String**: Encoded directly to base64 * **Object**: Automatically converted to JSON string, then encoded to base64 ### Response Returns the base64-encoded string directly (not wrapped in an object). ```text theme={null} "Y29uc29sZS5sb2coImhlbGxvIHdvcmxkISIpOw==" ``` ### Example Usage ```json Encode Text theme={null} { "function": "base64-encode", "params": "Hello World!" } ``` ```json Encode Object theme={null} { "function": "base64-encode", "params": { "name": "John Doe", "email": "john@example.com" } } ``` ```text In Flow theme={null} HTTP Request: Get file content ↓ Functions: Encode for transmission Function: base64-encode Input: (http_request.response) ↓ HTTP Request: Send encoded data ``` ### Common Use Cases Encode files before sending to APIs ``` Get file from storage Encode to base64 Send in HTTP request body API receives and decodes ``` Store binary content as text ``` Receive binary file Encode to base64 Store in database or KV storage Decode when retrieved ``` Create data URLs or embed in JSON ``` Get image file Encode to base64 Create data URL with encoded content Use in HTML or email ``` **Auto-Stringify**: Objects are automatically converted to JSON before encoding. No need to manually stringify objects. *** ## base64-decode Decode base64-encoded data. Automatically detects and parses JSON objects in the decoded output. ### Parameters Takes a single input value directly: The base64-encoded string to decode. ### Response Returns decoded data directly (not wrapped in an object). The return type depends on the decoded content: * **Object**: If the decoded string is valid JSON, returns the parsed object * **String**: If the decoded string is plain text, returns the string ```json theme={null} // Example 1: Decodes to plain text "Hello World!" // Example 2: Decodes to object (auto-parsed JSON) { "name": "John Doe", "email": "john@example.com" } ``` ### Example Usage ```json Decode Text theme={null} { "function": "base64-decode", "params": "SGVsbG8gV29ybGQh" } ``` ```json Decode Object (Auto-Parse) theme={null} { "function": "base64-decode", "params": "eyJuYW1lIjoiSm9obiBEb2UifQ==" } ``` ```text In Flow theme={null} Trigger: Receive base64-encoded data ↓ Functions: Decode data Function: base64-decode Input: (trigger.encoded_data) ↓ Agent: Process decoded content (automatically parsed if JSON) ``` ### Common Patterns Decode files received from APIs ``` HTTP Request: Receive base64 file Decode from base64 Store or process decoded file ``` Decode stored base64 data ``` Get base64 string from storage Decode to original format Use decoded content ``` **Auto-Detection**: This function automatically detects if the decoded string is valid JSON and parses it into an object. You don't need to specify the output format or manually parse JSON. *** ## json-xml Transform JSON data to XML format. Useful for integrating with systems that require XML. ### Parameters The JSON data to convert to XML. Can be: * **Object**: Converted directly to XML * **String**: Parsed as JSON first, then converted * **Buffer**: Converted to string, parsed as JSON, then converted XML conversion options: * `useXmlJs` (boolean): Use xml-js library instead of custom converter * `compact` (boolean, default: false): Produce compact XML format * `spaces` (number, default: 0): Number of spaces for indentation * `fullTagEmptyElement` (boolean): Use full tags instead of self-closing tags * `indentCdata` (boolean): Indent CDATA sections * `indentAttributes` (boolean): Print attributes on multiple lines * `ignoreDeclaration` (boolean): Omit XML declaration * `ignoreInstruction` (boolean): Omit processing instructions * `ignoreAttributes` (boolean): Omit element attributes * `ignoreComment` (boolean): Omit comments * `ignoreCdata` (boolean): Omit CDATA sections * `ignoreDoctype` (boolean): Omit DOCTYPE declaration * `ignoreText` (boolean): Omit text content ### Response Returns XML as a string with XML declaration. ```xml theme={null} John30 ``` ### Example Usage ```json Convert JSON to XML theme={null} { "function": "json-xml", "params": { "json": { "customer": { "name": "John Doe", "email": "john@example.com", "orders": [ {"id": "001", "total": 150}, {"id": "002", "total": 200} ] } }, "options": { "compact": true, "spaces": 2 } } } ``` ```json Compact Format theme={null} { "function": "json-xml", "params": { "json": { "QuestionAnswer": [ { "QuestionCd": "MOTOR_NCB", "Explanation": "1000" } ] }, "options": { "compact": true } } } ``` ```text In Flow theme={null} Agent: Generate order data (JSON) ↓ Functions: Convert to XML Function: json-xml JSON: (agent.order_data) Options: {compact: true} ↓ HTTP Request: Send XML to SOAP API ``` ### Common Use Cases Send data to XML-only systems ``` Collect data in JSON format Convert to XML Send to SOAP API or legacy system ``` Generate XML files for export ``` Build data structure in JSON Convert to XML with proper formatting Store as .xml file ``` Generate insurance industry standard formats ``` Build ACORD data structure Convert to XML with compact option Submit to insurance systems ``` **XML Declaration**: The output always includes XML version declaration at the beginning. Use ignoreDeclaration option set to true to omit it. *** ## xml-json Convert XML to JSON format. This is the reverse operation of json-xml, useful for parsing XML responses from APIs. ### Parameters The XML string to convert to JSON. XML parsing options: * `useXmlJs` (boolean): Use xml-js library instead of custom parser * `compact` (boolean, default: false): Produce compact JSON format * `trim` (boolean): Trim whitespace from text content * `sanitize` (boolean): Replace special characters with HTML entities * `nativeType` (boolean): Convert numeric/boolean strings to native types * `nativeTypeAttributes` (boolean): Convert attribute values to native types * `addParent` (boolean): Add parent property to each element * `alwaysArray` (boolean | array): Force elements to be arrays * `alwaysChildren` (boolean): Always generate elements property * `instructionHasAttributes` (boolean): Parse processing instructions as attributes * `captureSpacesBetweenElements` (boolean): Capture whitespace between elements * `ignoreDeclaration` (boolean): Skip XML declaration * `ignoreInstruction` (boolean): Skip processing instructions * `ignoreAttributes` (boolean): Skip element attributes * `ignoreText` (boolean): Skip text content * `ignoreComment` (boolean): Skip comments * `ignoreCdata` (boolean): Skip CDATA sections * `ignoreDoctype` (boolean): Skip DOCTYPE declaration ### Response Returns a JSON string (you may need to parse it in subsequent steps). ```json theme={null} "{\"Response\":{\"Result\":{\"Status\":\"Success\"}}}" ``` ### Example Usage ```json Parse XML Response theme={null} { "function": "xml-json", "params": { "xml": "Success", "options": { "compact": true, "ignoreDeclaration": true } } } ``` ```json Parse SOAP Response theme={null} { "function": "xml-json", "params": { "xml": "(http_request.response)", "options": { "compact": true, "trim": true, "nativeType": true } } } ``` ```text In Flow theme={null} HTTP Request: Call SOAP API ↓ Functions: Parse XML response Function: xml-json XML: (http_request.response) Options: {compact: true, trim: true} ↓ Agent: Process JSON data ``` ### Common Use Cases Convert SOAP API responses to JSON ``` Call SOAP API endpoint Receive XML response Convert to JSON Process as structured data ``` Parse XML from legacy systems ``` Receive XML from legacy system Convert to JSON Process with modern tools ``` Process XML files ``` Read XML file Parse to JSON Transform and process data ``` **Return Type**: This function returns a JSON string, not a parsed object. You may need to use JSON.parse() or another step to convert the string to an object for further processing. *** ## Handlebars Template Use Handlebars templating engine to create dynamic templates with variables that are replaced at runtime. ### Parameters The Handlebars template string with variables in double curly braces. The data object containing values to replace in the template. ### Response Returns the rendered template as a string. ```text theme={null} "Hello John Doe, your order #12345 has been confirmed!" ``` ### Example Usage ```json Email Template theme={null} { "function": "handlebars", "params": { "template": "Hello {{customer.name}}, order {{order.id}} confirmed!", "variables": { "customer": { "name": "John Doe" }, "order": { "id": "12345" } } } } ``` ```json Conditional Template theme={null} { "function": "handlebars", "params": { "template": "{{#if premium}}Premium{{else}}Standard{{/if}} customer {{name}}", "variables": { "name": "John", "premium": true } } } ``` ```text In Flow theme={null} Agent: Gather customer data ↓ Functions: Generate personalized email Function: handlebars Template: Email body with {{variables}} Variables: (agent.customer_data) ↓ HTTP Request: Send email ``` ### Template Features Insert dynamic values ```handlebars theme={null} Hello {{name}}, you have {{count}} new messages. ``` Access nested object properties ```handlebars theme={null} {{user.profile.firstName}} {{user.profile.lastName}} Email: {{user.contact.email}} ``` Show content based on conditions ```handlebars theme={null} {{#if isPremium}} Thank you for being a premium member! {{else}} Upgrade to premium for more features. {{/if}} ``` Iterate over arrays ```handlebars theme={null} Your orders: {{#each orders}} - Order {{this.id}} {{/each}} ``` ### Common Use Cases Create personalized emails ``` Email template with variables + Customer data = Personalized email content ``` Generate dynamic reports ``` Report template + Data from multiple sources = Formatted report ``` Create dynamic notifications ``` Message template + Event data = Personalized notification ``` *** ## Mapping Use JSONPath expressions to select, transform, and map data into new structures. Powerful for data transformation and restructuring with support for filters, array operations, and conditional selection. ### Parameters The source data object to query and transform. JSONPath expression(s) defining how to map the data: * **String**: Single JSONPath query * **Array**: For array transformations with mapping * **Object**: Map of output keys to JSONPath expressions Optional lookup data accessible as variables in path expressions for filtering and conditional selection. ### Response Returns the mapped/extracted data in the specified structure. ```json theme={null} { "customerName": "John Doe", "orderTotal": 150.00, "itemCount": 3 } ``` ### Path Syntax Access properties using JSONPath ```javascript theme={null} "$.user.name" // Get user.name "$.order.total" // Get order.total "$.items[0].price" // Get first item price "$" // Get root object ``` Combine values using pipe delimiter ```javascript theme={null} "$.firstName|' '|$.lastName" // Join with space // Result: "John Doe" "$.street|', '|$.city" // Join with comma // Result: "123 Main St, San Francisco" ``` Transform arrays with custom mapping ```javascript theme={null} [ "$.items", // Source array path { id: "$.id", name: "$.name" }, // Mapping for each item { merge: false } // Options ] ``` Create new object structure ```javascript theme={null} { userName: "$.user.name", userEmail: "$.user.email", orderTotal: "$.order.total" } ``` Filter using external variables ```javascript theme={null} // In path: "$.items[?(@.id === targetId)]" // In lookupData: { targetId: "abc123" } // Result: Item where id equals abc123 ``` Search recursively through nested structures ```javascript theme={null} "$..price.duty" // Find duty in any price object "$..covers..limits" // Find all limits in any covers ``` ### Example Usage ```json Simple Object Mapping theme={null} { "function": "mapping", "params": { "data": { "user": { "firstName": "John", "lastName": "Doe" }, "order": { "total": 150.00, "items": [1, 2, 3] } }, "path": { "customerName": "$.user.firstName|' '|$.user.lastName", "orderTotal": "$.order.total", "itemCount": "$.order.items.length" } } } ``` ```json Array Transformation theme={null} { "function": "mapping", "params": { "data": { "products": [ {"id": "p1", "name": "Phone", "price": 599}, {"id": "p2", "name": "Laptop", "price": 1299} ] }, "path": [ "$.products", { "productId": "$.id", "productName": "$.name", "productPrice": "$.price" }, { "merge": false } ] } } ``` ```json Filter with Lookup theme={null} { "function": "mapping", "params": { "data": { "orders": [ {"id": "001", "status": "completed"}, {"id": "002", "status": "pending"} ] }, "path": "$.orders[?(@.status === targetStatus)]", "lookupData": { "targetStatus": "completed" } } } ``` ```text In Flow theme={null} HTTP Request: Get complex API response ↓ Functions: Extract and restructure data Function: mapping Data: (http_request.response) Path: {custom mapping definition} ↓ Agent: Use clean, structured data ``` ### Common Use Cases Transform API responses to your format ``` External API format (complex) → Map to internal format (simplified) → Use in your system ``` Extract specific fields from complex objects ``` Large nested object → Extract only needed fields → Simplified object for processing ``` Convert between data formats ``` Source format A → Map fields to format B → Compatible with target system ``` Select data based on conditions ``` Large dataset → Filter by criteria using lookupData → Only matching records ``` **JSONPath Plus**: This function uses the jsonpath-plus library with full support for complex queries, filters, recursive descent, and array operations. *** ## secret-key-get-node Retrieve secret values stored in QuivaWorks' secret manager. Use this to securely access API keys, tokens, and other sensitive configuration. ### Parameters The name/identifier of the secret to retrieve. ### Response Returns an object with either a value property (on success) or an error property (if not found). ```json theme={null} // Success { "value": "sk_live_abc123..." } // Not Found { "error": "stripe_api_key not found" } ``` ### Example Usage ```json Get API Key theme={null} { "function": "secret-key-get-node", "params": { "key": "stripe_api_key" } } ``` ```json Get Database Password theme={null} { "function": "secret-key-get-node", "params": { "key": "postgres_password" } } ``` ```text In Flow theme={null} Functions: Get API key from secrets Function: secret-key-get-node Key: "external_api_key" ↓ HTTP Request: Call external API Authorization: Bearer (functions.value) ↓ Process API response ``` ### Error Handling Always check for the error property in the response to handle missing keys gracefully: ```javascript theme={null} // In your flow logic if (response.error) { // Key not found console.log(response.error); } else { // Use response.value const apiKey = response.value; } ``` ### Common Use Cases Securely access API keys for external services ``` Get secret API key Use in HTTP request headers Keep key secure, never in code ``` Retrieve database credentials ``` Get database password Connect to database Execute queries securely ``` Access service tokens ``` Get OAuth token Get webhook secret Use for integrations ``` **Security Best Practice**: Never hardcode secrets in flows. Always use the secret manager and retrieve secrets at runtime using this function. Secrets are stored in the quiva-secrets KV bucket. *** ## function-invoke Invoke other QuivaWorks functions programmatically from within your flow. Useful for orchestrating complex workflows and modular function composition. ### Parameters Type of invocation: * `"Async"`: Fire and forget (returns immediately, doesn't wait for result) * `"RequestResponse"`: Synchronous (waits for function to complete and returns result) Input data to pass to the invoked function. Structure depends on the target function's requirements. The identifier/name of the function to invoke. ### Response Returns an object. Structure depends on the invoked function and invocation type. ```json theme={null} // For RequestResponse { "result": "...", "status": "success" } // For Async { "invoked": true, "function": "function-name" } ``` ### Example Usage ```json Synchronous Invocation theme={null} { "function": "function-invoke", "params": { "invocation_type": "RequestResponse", "payload": { "data": "process this", "options": { "validate": true } }, "subject": "data-processor-function" } } ``` ```json Async Invocation theme={null} { "function": "function-invoke", "params": { "invocation_type": "Async", "payload": { "notification": "send email", "recipient": "user@example.com" }, "subject": "email-sender-function" } } ``` ```text In Flow theme={null} HTTP Request: Receive webhook ↓ Functions: Invoke data processor Function: function-invoke Type: RequestResponse Payload: (webhook.data) Subject: "webhook-processor" ↓ Functions: Async notification Function: function-invoke Type: Async Subject: "notification-sender" ``` ### Invocation Types Wait for function completion and get result ``` Invoke function Wait for execution Receive result Continue with result ``` **Use when**: You need the result to continue processing Invoke function without waiting ``` Invoke function Return immediately Function runs in background Continue without waiting ``` **Use when**: Result not needed, or for triggering side effects ### Common Use Cases Chain multiple functions together ``` Main flow receives request → Invoke validation function → Invoke processing function → Invoke notification function ``` Break complex logic into reusable functions ``` Create specialized functions Invoke them as needed Reuse across multiple flows ``` Trigger long-running operations ``` Receive request Invoke async background function Return immediate response Background task completes later ``` **Implementation**: Uses QuivaWorks SDK func.invoke() method. Ensures proper function calling within the platform's execution environment. *** ## sftp Upload files to SFTP servers securely. Supports both password and key-based authentication with configurable timeouts. ### Parameters SFTP server hostname or IP address. SFTP server port number (typically 22). Username for authentication. The file contents to upload (as a string). Remote file path where the file should be uploaded. SSH private key for key-based authentication. Alternative to password authentication. Password for password-based authentication. Alternative to key-based authentication. SSH algorithm configuration. Specify allowed server host key algorithms. ```json theme={null} { "serverHostKey": ["ssh-rsa", "ssh-ed25519"] } ``` Connection timeout in milliseconds. Default is 10 seconds. File transfer timeout in milliseconds. Default is 30 seconds. ### Response Returns a success message string on completion, or throws an error on failure. ```text theme={null} "File uploaded successfully" ``` ### Example Usage ```json Password Authentication theme={null} { "function": "sftp", "params": { "host": "sftp.example.com", "port": 22, "username": "ftpuser", "password": "secure_password", "fileContents": "Invoice data here...", "filePath": "/invoices/2025/invoice_001.txt" } } ``` ```json Key-Based Authentication theme={null} { "function": "sftp", "params": { "host": "secure-sftp.example.com", "port": 22, "username": "deploy", "privateKey": "-----BEGIN RSA PRIVATE KEY-----...", "fileContents": "(file_data)", "filePath": "/production/app.zip", "connectionTimeout": 15000, "transferTimeout": 60000 } } ``` ```json With Custom Timeouts theme={null} { "function": "sftp", "params": { "host": "sftp.company.com", "port": 2222, "username": "backup", "password": "backup_pass", "fileContents": "(backup_data)", "filePath": "/backups/daily/backup.sql", "connectionTimeout": 20000, "transferTimeout": 120000, "algorithms": { "serverHostKey": ["ssh-rsa"] } } } ``` ```text In Flow theme={null} Agent: Generate report file ↓ Functions: Upload to SFTP Function: sftp Host: sftp.example.com FileContents: (agent.report) FilePath: /reports/daily/report.pdf ↓ Functions: Send notification Message: "Report uploaded successfully" ``` ### Timeout Errors The function provides specific error messages for timeout scenarios: If server takes too long to accept connection: ``` Error: "Server took too long to connect. Make sure the host and the port are correct." ``` **Solutions**: * Verify host and port are correct * Check network connectivity * Increase connectionTimeout if server is slow If file upload takes too long: ``` Error: "File took too long to send." ``` **Solutions**: * Increase transferTimeout for large files * Check network bandwidth * Verify server is accepting uploads ### Authentication Methods Use username and password ```json theme={null} { "username": "user", "password": "secure_password" } ``` **When to use**: Simple setups, testing Use SSH private key ```json theme={null} { "username": "user", "privateKey": "-----BEGIN RSA PRIVATE KEY-----..." } ``` **When to use**: Production environments, automated systems, enhanced security ### Common Use Cases Deliver files to partners or systems ``` Generate report/export Upload via SFTP Partner retrieves file ``` Send backups to remote storage ``` Create backup Upload to SFTP backup server Verify upload success ``` Exchange data with external systems ``` Export data from your system Upload to partner's SFTP Partner processes file ``` Deploy files to remote servers ``` Build application files Upload to production SFTP Trigger deployment process ``` **Connection Management**: The SFTP connection is automatically closed after the upload completes or if an error occurs. No manual cleanup required. **Large Files**: For large files, increase the transferTimeout parameter appropriately. As a guideline, allow approximately 1 second per MB plus overhead. *** ## Best Practices base64-decode automatically parses JSON - leverage this for cleaner flows Check for error properties in responses, especially with secret-key-get-node Use Handlebars for emails, notifications, and reports with the variables parameter Use Mapping with JSONPath for powerful data transformations Prefer key-based authentication over passwords for production SFTP Always use secret manager for sensitive data - never hardcode secrets Use function-invoke for modular, reusable flow architectures Configure SFTP timeouts based on file sizes and network conditions *** ## Migration Notes If you're updating existing flows that use these functions, note these breaking changes: **Breaking Changes:** * **base64-encode**: Remove encoding parameter - pass data directly * **base64-decode**: Remove output\_encoding parameter - auto-detection enabled * **handlebars**: Rename data parameter to variables * **json-xml**: Restructure to use json and options object instead of separate parameters * **secret-key-get-node**: Rename key\_name to key, expect simplified response structure *** ## Next Steps Real-time event processing Fast key-based storage Store large files Using Functions in flows