Ed Vincent Chan

BHP

[ Data science - MEX Data Science ]

Data Science

• Explored and re-ran some of the prospectivity models that were migrated into Dataiku

Data Engineering

• Created an integration workflow to ingest model run results in Dataiku into the data platform
• Wrote a script for rebuilding a DynamoDB table using the available information from the raster files located in the S3 bucket
• Created functions for establishing connection from inside Dataiku towards various resources in AWS and Azure
• Wrote scripts for performing data cleaning and transformation on geochemical CSV data
• Converted analytical scripts written in Python and R into user-friendly and reusable application workflows and visual recipes inside Dataiku
• Converted machine learning scripts written in Python into user-friendly and reusable application workflows inside Dataiku
• Improved and generalized scripts for processing data to handle different kinds of geochemical data inputs
• Wrote scripts for performing various imputation, scaling, clustering, processing, analysis, transformation, classification, and visualizations on geochemical and geophysical data
• Created and deployed an Azure Function App for automating ingestion of data from Dataiku model run results into the data platform
• Wrote scripts for performing updates on a CosmosDB table
• Developed a custom plugin for establishing a connection from an AWS + S3-based data platform to Dataiku DSS through the Python API
• Developed a custom script to establish a connection and publish data records into ArcGIS online through its Python API
• Created a script for performing geospatial filtering on data records by specifying a bounding box or an area extent as described by a shapefile
• Created a custom geochronology toolbox visual application inside Dataiku using the tools included in IsoplotR
• Wrote functions for extracting important metadata from shapefiles and raster files
• Created dashboards for displaying results, metrics, and visualizations generated after running analytical scripts
• Packaged Python and R scripts into functions that can be reused in different workflows
• Developed a custom webapp using HTML, CSS, JavaScript, and Python (Flask) inside Dataiku for visualizing Hyperspectral images
• Developed an interactive plots widget using Python for masking points in a dataset
• Created scripts for reading, processing, and writing large datasets to/from S3 bucket

DevOps

• Provided support for resolving vulnerability issues in some Dataiku AWS resources
• Created functions for accessing data from various Azure resources like BlobStorage and CosmosDB to support migration of environment from AWS to Azure
• Handled various maintenance tasks to support the Dataiku DSS platform
• Resolved reported frontend and backend package security vulnerabilities in installed application dependencies
• Performed component version upgrades on applications by updating Terraform configuration files
• Debugged MongoDB and ElasticSearch application services through port-forwarding
• Updated Gitlab configuration files to resolve pipeline errors
• Performed deployment of application updates through the Azure CLI
• Updated dockerfile configurations to resolve various application issues

Testing and Documentation

• Provided support for and wrote documentation about various workflows written in FME
• Wrote automated checks for validating the results of the ingestion from the DynamoDB tables into the data platform
• Wrote documentations containing specifications and instructions of the geochemical and geophysical data analytics application workflows that were developed
• Created video tutorials for demoing how to use the geochemical and geophysical data analytics application workflows
• Created task scoping and estimates documentations for various projects
• Integrated automated checking of package vulnerabilities into Gitlab pipeline

BHP

[ Data platform application - MEX OSDU ]

Data Engineering

• Created functions for accessing the APIs of the data platform used to store mining data
• Wrote scripts for extracting metadata from different kinds of mining data
• Built custom schemas to store metadata extracted from different kinds of mining data
• Built workflows used for ingesting raw data into the data platform
• Created schemas and functions used in extracting data and ingesting records from a DynamoDB table into the data platform
• Created schemas and functions used in extracting relevant information from various Dataiku objects and ingesting it into the data platform
• Integrated the ingestion workflow with a Lambda function to automate ingestion to the data platform

Testing

• Wrote automated tests for validating the results of the functions in the SDK
• Performed manual QAQC of records ingested into data platform

BHP

[ Raster processing application - MEX Wormhole and Unearth4D ]

Data Engineering

• Created a python engine package that runs a workflow using the provided inputs and generates the final outputs to be displayed in the application frontend based on the scripts in the original notebooks

Web

• Created APIs using Django to obtain inputs from and pass engine outputs back to the application
• Integrated Celery to create a task queue for accomplishing the required engine model processes

Backend

• Integrated the engine functions with the data platform such that it will use the data platform as the input source and ingest the outputs back into it

Testing

• Wrote automated tests for validating the engine APIs

Mining Exploration Company

Web

• Added new features to the existing data platform application using Python with the Flask framework
• Updated application frontend with new functionalities using HTML, CSS, and JavaScript

Backend

• Managed existing databases by working with SQL scripts and queries
• Configured the Elasticsearch functionality of the application to properly display results from a search
• Wrote a scraping script that extracts geometry metadata from a web API

Testing

• Wrote automated tests using Pytest for web application features, functions, and API

DevOps

• Configured Docker and Terraform scripts to manage deployment of existing applications

MedTech Research Survey Application

Web

• Added a new feature to the existing web application using Python with the Django framework
• Implemented new frontend features using CSS, TypeScript, and ReactJS
• Added a new feature allowing users to receive an email containing the consent form of the surveys

Testing

• Wrote automated tests using Pytest for the Django models, views, and serializers of the application

Oil Company

Data Engineering

• Created equivalent functions in Python to perform calculations originally done through Excel formulas
• Contributed to creating a Python engine to be used for the application being built by writing functions that perform various computations, validations, and transformation on the data
• Implemented output error handling for the results of the engine calculations

Web

• Created APIs using Django to pass engine outputs to the application
• Added a new feature to the web application using Python with the Django framework
• Implemented minor frontend features in the application

Testing

• Wrote tests for validating the results of the engine functions used to build the main output dataset
• Wrote automated tests for the validating the application APIs