All Posts By

Angela Worsley

FME 2020.2 Delivers a Sneak Preview of 2021

The release of FME 2020.2 delivers some usability enhancements and additional format support. It also gives a sneak preview into FME’s 2021 release.   The Locus team reviews the highlights below.

FME Server

FME Server automations now have the ability to determine how your automations will run if the trigger fails by adding a failure outport port to each trigger. Reducing the need for you to watch your workflow as actions are taken automatically if the trigger fails.

FME Data Express and FME AR

Using the FME Data Express app to run and monitor FME Server jobs and Server apps from your mobile device, you now have the ability to search repositories using the new search bar, so you can easily find what you are looking for. Additionally, a new satellite map is now available in the location picker when selecting the geometry parameter and improvements have been made to the map scale and zoom level. Enhancements to FME AR include adding a compass so you can see your model’s orientation as you view it in the ‘real world’ and stabilization of geolocated models and better anchor handling. All making for a better experience and giving you more control.

New Application Support

With each FME release comes additional application support: With the 2020.2 release comes support for the following applications:

ArcGIS portal

Further to the release of the ArcGISOnlineConnector earlier in the year, you can now use this to access your ESRI ArcGIS Portal as a file storage service and access your stored ArcGIS content to upload, update, download, delete, or list information about your items using your ArcGIS credentials. Giving you the ability to manage your ArcGIS Portal data and automate your processes.

Shopify

Safe Software like to experiment and listen to customer feedback and ideas, which they can look to support new applications. At the request of some customers, Shopify is now in the Safe Lab (lab development stage) and can be used in FME by customers who use Shopify as their e-commerce platform. Simply by using the Shopify API, managing the backend data for your business can be easily done with the ability to automatically update your product catalogue, manage inventory and upload or update products. With FME you can easily sync between other applications, while automating your workflow.

Sneak Peak at the 2021.0 Release: Event Streaming Services

In the 2021 release FME will make it easier to integrate IoT and streaming data, creating a better user experience for event stream processing capability. This enables you to process and integrate continuous, real-time data of any volume by processing the data in your workflow before storing it. Providing a great alternative to bulk/batch workflows.

Support is now available on the FME hub for the following:

Article Author: Pip Norris, FME Data Solutions, Locus

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

Auckland Council Delivers a Single-Automated Solution for its Digital RFS App.

Auckland Council is the local authority that is responsible for all local government decisions and responsibilities in the Auckland region. In New Zealand, central government has created a local government structure for every region and district in New Zealand.

The Background

Auckland Council is responsible for managing thousands of assets across New Zealand’s most populated city.  From rubbish bins to leisure centres and everything in between. Each one of these assets must be maintained with the help of various contractors, often relying on reported problems from the public to trigger this work to take place. Locus was recently engaged to assist in a project that automates the end-to-end processing of these requests for service, significantly reducing administrative overhead, and improving customer service.

Locus’ role in this project was to develop a data processing model in FME that could manipulate dozens of GIS asset and contextual feature classes with over a million records, into a useable format. The transformed data allows the public and customer services agents to select assets in a web map interface and pass attributes through to SAP and other business systems for processing. This process is now automated with FME Server to ensure the Report A Problem web application for asset maintenance is always up to date with the latest assets from the enterprise system.

The Challenge

The Digital RFS (Request for Servive)  app needed a single automated ETL solution that would remove data inconsistencies, manual processing and maintain the latest assets. The data had its quirks, repositories, update schedules and owners. Geometries were stored in GIS while most attributes were stored in SAP, and refreshed every week using FME server. Only the latest information would be useful and so the consolidated data needed to be updated automatically after the weekly refresh.

“FME never ceases to amaze with how it can simplify and create order out of extremely complicated data processing operations. It’s simple enough to dive into for anyone to have a go. However, we’ve certainly appreciated having Kieran O’Donnell from Locus engaged on our projects and make full use of its capabilities”

Auckland Council

The Results

The following key project deliverables/outcomes were achieved:

  • A single solution to read, process, publish data for the Digital RFS solution
  • Reads 20 enterprise feature classes from Auckland Council’s SDE
  • ‘Cookie-Cut’ process to identify all feature intersections and assigns the classification based on the asset hierarchy
  • FME Server used to process data into a single Query Layer, Query Table, 19 reference layers
  • FME Server manages a python script to automate service publishing to keep ArcGIS Online hosted service in-sync with the data on the premises.

The Future

Auckland Council plans to refresh the data structure soon and will continue to utilise the power of FME to update the data.  There are plans to integrate more data sources and widen the scope of the solution.

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

Integrating FME with ArcGIS Online

ArcGIS Online is a great tool for hosting your data online, and building web maps or apps which you can use to share data with your users. Using FME, you can make sure you’re getting the most from your ArcGIS Online subscription by building some really great workflows to keep your data up to date, pull data down into your organisation, or even administer your ArcGIS Online account. 

ArcGIS Online Feature Service Writer 

The ArcGIS Online Feature Writer is one of many ways you can use FME to integrate with ArcGIS Online. Using this writer, you can create, update or remove data which is in your ArcGIS Online organization. For many organisations publishing datasets to ArcGIS Online creates a bit of a data management headache, as Hosted Feature Services are duplicates of data which is maintained internally.  

Using FME however, you can create a simple process to read from your internally maintained dataset, and write directly to the feature service. By running this process on a regular basis, the data hosted in ArcGIS Online will stay up to date and you can have confidence in all your data without a burdensome process to maintain. 

ArcGIS Online Feature Service Reader 

Using the ArcGIS Online Reader, you can also access any Hosted Feature Service you have published. By adding this into an FME Workspace, you can read the data like any other format. This is particularly useful if you have projects utilising ESRI’s Collector application, and FME even allows you to download the attachments stored against each feature. 

Just like using the Feature Service Writer, you can keep two datasets in-sync without needing to constantly process the data yourself. In this case, the master dataset is held in the ArcGIS Online Feature Service, and an internal dataset is the replica which you use to display in your in-house GIS systems. 

ArcGIS REST API 

Another way to integrate FME with ArcGIS Online is using the REST API. This is slightly more advanced, so users who are unfamiliar with accessing APIs in FME should check out our FME and the art of the API blog article. We’ve also put together a sample workspace you can use to get started. Some instructions are included in the workspace to set this up to work with your own ArcGIS Online account. 

The ArcGIS REST API has a wealth of functions you can use to perform geospatial, mapping and administrative functions using your ArcGIS Enterprise organisation or ArcGIS Server. Using FME, you can make HTTP Calls to the ArcGIS REST API to perform these functions in an automated fashion. 

In the sample workspace, the process takes you through a fairly simple workflow to perform a query on your ArcGIS Online organisational account. Like most APIs you need to authenticate with ArcGIS Online first to fetch a token, in order to validate you have permissions to perform this function. 

Next, the workspace performs another request to the REST API, querying the portal with the user parameter you have set. This could be for content owned by a specific user, containing a certain tag, or a combination of criteria – the potential is endless. After making the request, the JSON is unpacked and split into individual records. In this case, the output is written to a spreadsheet of items which do or don’t contain tags, but again you can perform any function you wish. Querying the organisation is a great way to keep on top of data or items hosted in ArcGSI Online, in order to ensure duplicates aren’t kept and metadata standards are adhered to over time. 

The functions aren’t just limited to querying the portal though. You can use FME to publish services, create new items, or migrate content from one user account to another. If you’re willing to spend a little time developing these processes, you can get some great functionality which will save you plenty of time and effort in administering your organisationalaccount, and ensure it’s always kept up to date. More information on the ArcGIS REST API. 

Article Author: Kieran O’Donnell, FME Data Solutions, Locus

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

Our FME 2020.0 Top Picks!

The latest version of FME is here, with some significant improvements to the 2020 release giving you greater performance, flexibility and accessibility to the power of FME.

We asked our Locus FME Certified technical team to take a closer look at some of the 2020.0 enhancements.

FME Server

Access to FME Server has never been easier, with new options for authentication and the ability to run jobs from a mobile device. This gives you the power to check on or run jobs remotely, monitor logs, or even build a data capture service via mobile. Download the FME Data Express mobile application now to get started.

FME Server Applications can now use authentication, giving you greater flexibility with security whilst still opening up the power of FME Server to a wider audience. This can be controlled on the application level, letting you choose which apps are shared with different individuals or even roles or groups.

A key enhancement to FME Server apps is also the ability for users to input geometry with a map picker as a published parameter. This enhances the user experience of running server apps, and opens your workspaces to an even wider audience by allowing non-technical staff to select areas of interest, straight out of the box.

 

FME Desktop

As always, FME Desktop has undergone significant performance enhancements. When upgrading your version of FME Desktop, always check to look for upgradeable transformers which will improve the efficiency of your workspace. The Statistics Calculator is roughly 100x faster than it’s last version, so it’s definitely worth upgrading sooner rather than later.

Two of the most used formats; Shapefile and Excel reader/writers are now lightning fast. These popular formats use new underlying data structures, so your workflows that connect to these formats will run more efficiently than ever.  After upgrading, when you open your workspaces that use these formats, make sure you update the reader/writers to the newer version.

You also now have the ability to apply custom connection line styles in your FME Workspaces to enhance the user experience when building up your workflows.

Like any release of FME there has been a series of new formats added for both Reading and Writing. Connect to Big Data systems, spatial data and 3D models, XML and JSON formats, and more.

New readers, writers, connectors, and transformers in FME 2020.0 include:

  • Azure and Google cognitive services for natural language processing and computer vision
  • CityJSON (Tech Preview)
  • Entwine Point Tile (EPT)
  • Google BigQuery
  • Mapbox Vector Tiles (MVT)
  • Microsoft DirectDraw Surface (DDS)
  • NIfTI (for MRI data)
  • OGC S-121 GML
  • PROJ transformers (leverage more coordinate systems by connecting to the PROJ reprojection library)
  • Revit (including floor plans and easy wall simplification)
  • Snowflake
  • UK MasterMap variants: Water, Highways, Sites, Topography
  • XSD-driven XML

Article Author: Kieran O’Donnell, FME Data Solutions, Locus

More about Locus …Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.

Ready to upgrade to FME 2019.1

With the release of 2019.1 on both FME Desktop and Server some impressive new enhancements deliver visual and performance based gains.

What’s new in 2019?

FME Server Automations: Released in 2019.0, Automations are a revolutionary new tool in FME Server where you can use a graphical builder to create event based workflows. You can build workflows by defining triggers and actions, such as receiving an email, watching a directory or using a schedule. Automations are revolutionary because you can build workflows that leverage all the power of FME Server, in an easy to use drag and drop interface.

 

Visual Preview: This feature allows you to preview your data from within a dockable pane inside FME Workbench, without the need to open a data inspector window. Our users find this particularly helpful when working with tricky logic and minor tweaks  to their workspace, and definitely makes building and debugging your workspaces more efficient.

Performance Enhancements: With all FME releases there are some enhancements to performance, but this version in particular brought some really great improvements. Shapefiles have had a redevelopment to become incredibly fast and efficient, which is great news for the most popularly used format. The Dissolver transformer has also been seriously enhanced, running as much as 75x faster than before.

New Formats: As always the number of formats supported in FME has increased, with this year the number reaching more than 450. Of note is a new native Revit reader, where you can read your Revit files directly without needing to use RVZ intermediate files. Some users will also be interested that there is now a 12d Reader released in beta, with expected support in FME 2020.

More about Locus…Locus is a team of business practitioners and FME technical experts ready to help you maximise the value of your business through better data management. As a Safe Software Platinum Partner and Value Added Reseller with offices in New Zealand and Australia, we hold the unique position of being engaged solely in the sale, support and servicing of FME products; this is our competitive advantage.

More about FME…Built by Safe Software and used by over 20,000 organisations worldwide, FME is a data translation and transformation tool for solving problems of data interoperability, without the need for coding. With support for 400+ formats and applications, FME is the integration platform that converts, transforms and automates data whenever and however it’s needed.