Monday, June 29, 2015

SSIS : Retry Extract on connection time.

Requirement: We pull AD Users through web service. The connection often times out. We would like to re-try connecting 3 times, if all three attempts fail continue with the ETL process without abend.
Send an email notification about the connection timeout.

Solution:

Part A

In the Extract Child Package -


 Create a for loop for data extract, which is initialized as follows –



Inside the for loop add on “On Success” and on “On Fail” script tasks apart from Data Flow Task which does the extract.

The “On Success” task sets vContinue variable to 0
The “On Fail” task increments vFail variable, if the loop fails three times, it sets vContinue variable to 0.



Go to On Error event handler for Data Flow task.

Create an empty On Error Event Handler.
Set Propagate system variable value to “False”.
Setting propagate to false will prevent  OnError from bubbling up to parent container.









Add a Send Mail Task with Precedence Constraint of @vFailCount>0 .

The solution so far will prevent the child package from failing and have it re-try 3 times before exiting.
The error will however still propagate up to parent package, to prevent parent package from failing follow the Part B of the solution.


Part B

In the parent package which calls this child package.
Go to the Execute Package Task for this Child Package.
Create an empty OnError Event handler and Set Propagate System variable to False.




End Result:
We have an extract package which connects to web service, if there are time outs it will retry 3 times before exiting. The extract package also sends email about timeout failure.
The extract package is called from a parent ETL package which ignores webservice connection failures and continues with other data load tasks.



Friday, June 12, 2015

SQL Server 2016 BI Enhancements

Database Engine
Updatable Non-Clustered Column Store Indexes.

R-Integration
R Predictive Analytic functions available via T-SQL.

Polybase Support
Polybase was available only in PDW, now to be available in SQL Server.
Polybase is a technology that allows querying of non-relational data in Hadoop Distributed File System using SQL.

SSIS

SSIS Designer to support previous versions.
Power Query is available as a Data Source, this would allow for Self Service ETL to Enterprise ETL conversion.
New Connectors available for
                Hadoop HDFS.
                JSON
                Azure Storage
                Oracle
                Teradata
ETL Package Templates

SSRS
Ability to create reports for Mobiles.
Support for more browsers.
New Chart types.
Connectors for Oracle, Teradata, SAP BW.
Parameter Enhancements.  
              
Development
BIDS and SSDT Unified in Visual Studio.

SSAS Multi-Dimensional
DBCC commands for detecting data issues.
Distinct count improvements.
Support for Netezza as data source.

SSAS Tabular
Support for Many to Many Relationships.
Bi-Directional filtering.
New DAX Functions – DATEDIFF, GEOMEAN, PERCENTILE, MEDIAN, MEDIANX etc.
Better Time Intelligence Support – Eliminates need for Date Table.
Partitioned table processing – Multiple partitions can be processed in parallel.


Wednesday, June 10, 2015

Query to Build Date Table

Query to build a date table.  Use it to create date table on fly in Power Pivot or Persist the results to a date dim table in your data mart.

It will run in SQL Server 2005 and higher.


WITH Dates AS
      (
            SELECT      CAST('20000101' AS DATE) AS [Date]
            UNION ALL
            SELECT      DATEADD(DAY, 1, [Date])
            FROM  Dates
            WHERE [Date] < '20251231'
      )
      SELECT      Date,
                  CONVERT(Varchar,Date,112) as DateID,
                  YEAR(Date) as Year,
                  DATEPART(Quarter,Date) as Quarter,
                  DATEPART(WEEK,Date) as Week,
                  DATEPART(WEEKDAY,Date) as DayofWeek,
                  MONTH(Date)as MonthNumber,
                  DateName(Month,Date) as MonthName,
                  DateName(WeekDay,Date) as DayOfWeekName,
                  Case WHEN DATEPART(WEEKDAY,Date) in (1,7) THEN 0 ELSE 1 END as WeekDayFlag
      FROM  Dates
      OPTION ( MAXRECURSION 10000 );