Software Engineering Stack Exchange
Software Engineering Stack Exchange

Questions tagged [etl]

Extract, Transform, Load - process in a database

50 questions
8
votes
3 answers

How to do data synchronisation between two applications?

Our client has a CRM application (let's call it A) and a custom built Windows application (VB.NET) (let's call it B) that directly writes to and reads from the CRM database. Application A is used (mainly) to manage clients and their orders.…
Kristof Claes
  • 3,100
  • 2
  • 21
  • 33
6
votes
2 answers

Disagreement Concerning Data Integration (I may not understand enterprise ETL tools)

I have been in an ongoing conversation concerning a project we are about to undertake at my place at work. The project concerns data integration. Our customers want to be able to integrate our data with theirs, either by sending data from their…
Josh
  • 321
  • 1
  • 2
  • 7
5
votes
2 answers

How can large data handling be a special requirement for job?

I chanced to see some job requirements of Oracle SQL programmer or C# developer jobs, who want a person who can handle terabytes of data in ETL or data migration. I have worked on data migration to move data from one system to the other. As a C#…
quantcoder
5
votes
1 answer

Difference between ESB and ETL

When should a ESB vs ETL tool be used? I have worked on ESB projects using Tibco Businessworks quite a few years ago. The message bus that we built used to consume messages from a source system , applying validation , perform transformation and send…
Punter Vicky
  • 875
  • 4
  • 11
  • 19
4
votes
2 answers

Designing an ETL with where there are a few points of entry

I'm trying to think of a scalable solution for my current system. The current system is 3 microscopes 1 processing machine 1. 60-100GB Files come from 2-3 microscopes every 30 minutes 2. That data is transferred to a (local) network mount of the…
user3145912
  • 49
  • 3
4
votes
2 answers

How should a data warehouse be maintained for a quickly changing schema

I am currently in a process of maintaining a data warehouse for a quickly growing start up company. There is a lot of reporting demands from the clients, and this is usually handled by a data warehouse we set up. However, unlike bigger more…
Yong Jun Kim
  • 151
  • 4
4
votes
2 answers

Is ReST useful in Read/Write Operations that involve over 100 Gig

I work in Healthcare and we use SAS to Extract and Transform medical and pharmacy claims data for use in downstream reporting applications. For a given Report Request(usually 40 are running at a time) we have to extract from various DBMS's and…
Charlie Bastnagel
  • 41
  • 2
4
votes
1 answer

What is the right way to process inconsistent data files?

I'm working at a company that uses Excel files to store product data, specifically, test results from products before they are shipped out. There are a few thousand spreadsheets with anywhere from 50-100 relevant data points per file. Over the…
Tahabi
  • 61
  • 1
  • 6
4
votes
1 answer

Designing a Content-Based ETL Process with .NET and SFDC

As my firm makes the transition to using SFDC as our main operational system, we've spun together a couple of SFDC portals where we can post customer-specific documents to be viewed at will. As such, we've had the need for pseudo-ETL applications to…
Patrick
  • 165
  • 1
  • 6
3
votes
1 answer

What are some design ideas for a data mapping and transformation application?

Here is a high level outline of the project: We frequently need to convert data from a new incoming system to our in house system (sort of a basic ETL process) We would prefer to do this dynamically, allowing an analyst to use a GUI to map source…
dpberry178
  • 141
  • 4
3
votes
2 answers

Translate data between inconsistently-matched data structures

How can my program best represent a translation between imperfectly-matched data structures? I am tasked with a one-way translation of data from one system to another. Both systems are established, I don't have the option of changing their data…
bignose
  • 191
  • 9
3
votes
1 answer

How is one or more aggregate function implemented in most SQL engines?

In the book Database Fundamentals, Silberschatz. It is explained that aggregate functions can be calculated on the march. This make sense. What it means is that for calculating the maximun, average or count the items in a set, you don't need to pass…
jgomo3
  • 336
  • 1
  • 12
3
votes
2 answers

Data warehouse architecture for mutating schema

I am setting up an ETL process and small data warehouse for querying the data in a few different dimensions. One issue is that the schema for the objects can mutate over time - mainly that some fields will be added and some removed. What are some…
Rex M
  • 231
  • 1
  • 8
3
votes
1 answer

Enterprise Wide Keys

I have for a long time been working on an ODS as well as Data Warehouse. Both are integrating a wide variety of data sources from stove pipe applications. One of the uses of the ODS is to provide data to other stove pipe applications. Imagine…
AaronLS
  • 206
  • 1
  • 9
2
votes
2 answers

Reading a large CSV file and then loading data to a DB

I have a Django application of 2 GB running and I need to receive a CSV file of more than 1 GB, read it and load the data to a PostgreSQL DB in IBM Cloud. The problem is that if I receive the file, it would have to be stored locally and I will…
Elvin Quero
  • 29
  • 1
1
2 3 4