Все вопросы: [etl]

61 вопросов

похожие теги: rhino-etl
2
голосов
5ответов
3578 просмотров

Инструменты ETL и инструменты сборки

Я знаком с инструментами автоматизированной сборки программного обеспечения (такими как Automated Build Studio).Теперь я смотрю на инструменты ETL. Одна вещь приходит мне в голову: я могу делать все, что могу, с инструментами ETL, используя инструмент сборки программного обеспечения.Инструмент...

0
голосов
4ответов
2814 просмотров

Альтернатива для задачи поиска в SSIS

Я работаю над решением SSIS для хранилища данных для извлечения суррогатных ключей соответствующих ключей приложения. Я использую задачу поиска SSIS, но проблема с этой задачей заключается в том, что она кэширует полную таблицу поиска в своей памяти.И размер моей таблицы поиска огромен, то есть ...

1
голосов
1ответов
1337 просмотров

Problem with Rhino-Etl and MySQL

I've been using Rhino-ETL for a little while and it's running pretty smooth. However I have a problem connecting to my MySQL DB. Rhino.Etl.Core.RhinoEtlException: Failed to execute operation Hornalen.Migration .Process.ReadMessagesFromDb: The type name 'MySql.Data.MySqlClient' could not be foun...

4
голосов
2ответов
3623 просмотров

Where is Pentaho Kettle's architecture?

Where can I find Pentaho Kettle architecture? I'm looking for a short wiki, design document, blog post, anything to give a good overview on how things work. This question is not meant for specific "how to" starting guides but rather a good view at the technology and architecture. Specific questi...

2
голосов
6ответов
2445 просмотров

Move data from one database to other with different data structure

How to move data from suppose mysql database to postgres database? Scenario: Two similar application. A user wants to switch from one application to other. But he had maintained certain data information in his previous appilaction which uses mysql database.When he switch his appliaction he has t...

3
голосов
3ответов
1388 просмотров

How do I keep a table synchronized with a query in SQL Server - ETL?

I wan't sure how to word this question so I'll try and explain. I have a third-party database on SQL Server 2005. I have another SQL Server 2008, which I want to "publish" some of the data in the third-party database too. This database I shall then use as the back-end for a portal and reporting s...

0
голосов
6ответов
1190 просмотров

ETL Tool for transfering old Firebird Database to a new organized Firebird Database

After looking at a lot of questions..i found no real answer for this. I redisigned an Database for our customer. With Microsoft Access i found a good Tool to get old table Data in my new well formed Database Structure. It is really easy but takes a lot of time (cause handling old Data with a lot...

2
голосов
3ответов
284 просмотров

Advice on how to write robust data transfer processes?

I have a daily process that relies on flat files delivered to a "drop box" directory on file system, this kicks off a load of this comma-delimited (from external company's excel etc) data into a database, a piecemeal Perl/Bash application, this database is used by multiple applications as well as...

1
голосов
2ответов
532 просмотров

Large scale ETL string lookups performance issues

I have an ETL process performance problem. I have a table with 4+ billion rows in it. Structure is: id bigint identity(1,1) raw_url varchar(2000) not null md5hash char(32) not null job_control_number int not null Clustered unique index on the id and non clustered unique index on md5hash S...

0
голосов
5ответов
256 просмотров

SQL Server 2005 loading data from an external server

Have a new project with the following setup and requirments:- My client has a MSSQL 2005 server (A) in their office. Their vendor has a MSSQL 2005 server (B) in another part of the world, which contains real-time transactional data. My client wants to load the data from (B) to (A) on a daily bas...

0
голосов
1ответов
87 просмотров

Problem regarding integration of various datasources

We have 4 datasources.2 datasources are internal and we can directly connect to the database.For the 3rd datasource we get a flat file (.csv) and have to pull in the data.4rth datasource is external and we cannot access it directly. We need to pull data from all the 4 datasources, run business r...

3
голосов
1ответов
11854 просмотров

Easiest way to import CSV into SQl Server 2005

I have several files about 5k each of CSV data I need to import into SQL Server 2005. This used to be simple with DTS. I tried to use SSIS previously and it seemed to be about 10x as much effort and I eventually gave up. What would be the simplest way to import the csv data into sql server? I...

11
голосов
3ответов
2558 просмотров

What are the required functionalities of ETL frameworks?

I am writing an ETL (in python with a mongodb backend) and was wondering : what kind of standard functions and tools an ETL should have to be called an ETL ? This ETL will be as general purpose as possible, with a scriptable and modular approach. Mostly it will be used to keep different databas...

1
голосов
1ответов
2558 просмотров

MapForce vs. Talend Open Studio

We have been using Talend 3.1 for a few months now. However, we are looking at possibly switching to the latest MapForce. Simply because it compiles to a .Net solution and we are otherwise a .Net house. That being said Talend is extremely easy to use and extend. The Talend jobs make it very easy ...

-1
голосов
2ответов
728 просмотров

How can I translate these sed and perl one-liners to informatica?

Duplicate: https://stackoverflow.com/questions/1259545/let-me-know-alternate-command-in-dos-for-following-sed-and-perl-commands-closed the following commands have unique implementation in unix box. Need to implement in informatica(etl tool). If not any windows solution for the same sed 's/^#...

194
голосов
12ответов
247986 просмотров

MySQL - Rows to Columns

I tried to search posts, but I only found solutions for SQL Server/Access. I need a solution in MySQL (5.X). I have a table (called history) with 3 columns: hostid, itemname, itemvalue. If I do a select (select * from history), it will return +--------+----------+-----------+ | hostid | i...

-1
голосов
3ответов
1360 просмотров

ETL as a transaction

For all the ETLs I have written so far, I have never made them transactions - i.e. if table 4 fails, roll everything back. What is the best practice in this regard? To "BeginTran + Commit" or not to "BeginTran + Commit" EDIT: I have one master package calling 4 other packages - is it possible ...

1
голосов
1ответов
784 просмотров

SSIS (missing) Pre-Build and Post-Build

For the warehouse work under progress, we have a single solution with multiple projects in it OLTP Database Project Warehouse Database Project SSIS ETL project After the SSIS project is built, I want to move the binaries (XML, really) from the Bin folder to "C:\AutomatedTasks\ETL.Warehouse\" ...

3
голосов
5ответов
4855 просмотров

informatica powercenter vs custom perl ETL job?

Most of my company uses powercenter informatica for Extract-Transform-Load type data move jobs between databases. However project I am on has a big custom Perl job with some Java thrown in for good measure to move data and trigger some other updates. There is talk of rewriting the thing to us...

1
голосов
1ответов
517 просмотров

Tracking what the MERGE command and its OUTPUT did

I am modifying a Type 2 dimension using the following (long) SQL statement: INSERT INTO AtlasDataWarehouseReports.District ( Col01, Col02, Col03, Col04, Col05, Col06, Col07, Col08, Col09, Col10, StartDateTime, EndDateTime ) SELECT Col01, Co...

6
голосов
3ответов
13495 просмотров

ETL tools... what do they do exactly? In laymans terms please

I have recently been exposed to some ETL tools such as Talend and Apatar and I was wondering what exactly the purpose/main goal of these tools is in laymans terms. Who primarily uses them and if you use them, how they are (from my understanding) better than just writing some type of scripts.

5
голосов
7ответов
6129 просмотров

Data extraction with Excel

I monthly receive 100+ excel spreadsheet from wich i take a fixed range and paste in other spreadsheet to make a report. Im trying to write a vba script to iterate my excel files and copy the range in one spreadsheet, but i havent been able to do it. Is there an easy way to do this?

1
голосов
3ответов
13722 просмотров

How to export text data from a SQL Server table?

I am trying to use the MS SQL Server 2005 Import/Export tool to export a table so I can import it into another database for archival. One of the columns is text so if I export as comma-delimited, when I try to import it into the archive table, it doesn't work correctly for rows with commas in tha...

3
голосов
1ответов
250 просмотров

Recording MySQL DELETE statements

We have a MySQL->Oracle ETL using Informatica that works great for all statements except DELETE. Unfortunately, the DELETE makes the record go away such that Informatica never sees it again to remove/expire it in Oracle. How have people gone about recording MySQL DELETE statements? The tabl...

1
голосов
6ответов
1053 просмотров

Integer zero, "0' will be ignored when upload to SQL Server

i have a page that allow user to upload an excel file and insert the data in excel file to the SQL Server. Now i have a small issue that, there is a column in excel file with values, such as "001", "029", "236". When it's insert to the SQL Server, the zero in front will be ignored in SQL, so the ...

1
голосов
3ответов
2228 просмотров

In SQL Server CDC with SSIS, which data should be stored for windowing (LSN or Date)?

I have implemented delta detection while loading data warehouse from transaction systems using an identity column or date-time column in source transaction tables. When data needs to be extracted next time, the maximum date-time value extracted last time is used in the filter of extraction query ...

1
голосов
3ответов
1351 просмотров

Loading data from SAS to Teradata - When is it ready?

When loading tables from SAS to Teradata, SAS loads the data (usually using the FASTLOAD facility) and then continues down the script. However, I often get critical errors because SAS says the data is loaded, but Teradata is still assembling the data within the table. So the data is in the data...

1
голосов
2ответов
485 просмотров

.Net Журнал событий

Я пытаюсь заставить новый журнал событий System.Diagnostics.Eventing работать в простом приложении .Net, прежде чем интегрировать его в свое приложение. Отработка На этой странице я создал манифест, создал простое приложение, которое запускает событие, и зарегистрировал поставщика, чтобы ув...

0
голосов
1ответов
855 просмотров

Как лучше всего повторно использовать бизнес-логику на страницах Informatica ETL и ASP.NET CRUD?

Я ничего не знаю об Informatica, но я ищу способы решить проблему дублирования бизнес-логики, которая используется для вставки и обновления записей в таблице. Проблема в том, чтобы сделать это эффективно. 1) У нас есть веб-страницы, которые вставляют, обновляют и удаляют записи по одной. 2...

1
голосов
1ответов
1664 просмотров

Выгрузка Excel в таблицу базы данных

Я ищу лучшее решение, позволяющее нашим пользователям загружать электронную таблицу XLS, чтобы их можно было использовать для заполнения таблиц в нашем хранилище данных (DW). Наши пользователи - активные пользователи бизнес-объектов (BO), и BO позволяет экспортировать их в XLS. Когда у них ес...