Decoding Epoch Time in FME: A Comprehensive Guide

Working with geospatial data often involves encountering different time formats. One common format is Epoch time, a system for tracking a point in time as a single number representing the seconds that have elapsed since the beginning of the Unix epoch (January 1, 1970, at 00:00:00 Coordinated Universal Time (UTC)). While Epoch time is useful for computers, it's not very human-readable.

This article will guide you through understanding and converting Epoch time to a human-readable date and time format using FME (Feature Manipulation Engine), a powerful data integration platform. We'll address a common issue encountered when working with Epoch time from sources like the USGS (United States Geological Survey) and provide a step-by-step solution.

The Challenge: Milliseconds vs. Seconds

A user encountered difficulty converting Epoch time from a USGS GeoJSON dataset using FME's DateTimeConverter. The data contained Epoch timestamps like 1650016756409 and 1649948565590, and attempts to convert them using format strings %s or %Es resulted in an "Invalid_input" rejection code.

The root cause? The Epoch time values were in milliseconds, while the DateTimeConverter with the %s and %Es formats expects seconds. This discrepancy is a common pitfall when working with Epoch time.

The Solution: Converting Milliseconds to Seconds

To resolve this, you need to convert the Epoch time from milliseconds to seconds before using the DateTimeConverter. Here's how you can do it using FME:

1. Identify the Attribute: Determine the attribute in your FME workspace that contains the Epoch time in milliseconds. Let's assume it's named datetime_epoch_milliseconds.

2. Use the ExpressionEvaluator or AttributeCreator: Both transformers can perform mathematical calculations. We'll use them to divide the millisecond value by 1000 to get the equivalent value in seconds.

   • ExpressionEvaluator: Add an ExpressionEvaluator transformer to your workspace. In the parameters, create a new attribute (e.g., datetime_epoch_seconds) and set its value to the following expression:

     datetime_epoch_milliseconds / 1000
     

   • AttributeCreator: Add an AttributeCreator transformer to your workspace. Create a new attribute (e.g., datetime_epoch_seconds) and set its value to the following expression:

     @Evaluate(@Value(datetime_epoch_milliseconds)/1000)
     

     The @Evaluate function ensures that the expression is evaluated, and @Value(datetime_epoch_milliseconds) retrieves the value of the datetime_epoch_milliseconds attribute.

3. DateTimeConverter: Now that you have the Epoch time in seconds, you can use the DateTimeConverter transformer to convert it to a human-readable format.

   • Connect the output of the ExpressionEvaluator or AttributeCreator to the DateTimeConverter.
   • Set the "Input Format" to %s or %Es (both represent Epoch seconds).
   • Set the "Output Format" to your desired date and time format (e.g., %Y-%m-%d %H:%M:%S for "YYYY-MM-DD HH:MM:SS"). Refer to the FME documentation for a complete list of format codes.
   • Set the "Input Attribute" to the name of the new attribute you created (datetime_epoch_seconds).
   • Set the "Output Attribute" to the name of the attribute where you want the human-readable date and time to be stored (e.g., datetime_human_readable).

Example FME Workflow

Here's a simplified representation of the FME workflow:

1. Feature Reader: Reads the GeoJSON data.
2. ExpressionEvaluator or AttributeCreator: Converts Epoch milliseconds to seconds.
3. DateTimeConverter: Converts Epoch seconds to a human-readable date and time format.
4. Feature Writer: Writes the transformed data.

Key Takeaways

• Epoch Time Units: Always verify whether your Epoch time values are in seconds or milliseconds.
• FME Transformers: The ExpressionEvaluator and AttributeCreator are versatile transformers for performing calculations.
• DateTimeConverter: This transformer is crucial for converting between different date and time formats in FME.
• Documentation is Your Friend: Refer to the official FME documentation for detailed information on transformers and format codes.

By understanding the nuances of Epoch time and utilizing FME's powerful transformers, you can seamlessly convert Epoch time to human-readable formats, enabling you to work effectively with geospatial data from various sources.