Download Freshdesk Data using Data Pipeline in Fabric runs forever

Questions
, ,
1

I apologize, my browser translator is not working and I don’t understand Dutch at all.

I am trying to setup a connection within my Fabric workspace to download a table into our Data Lake. I’m assuming I will need to use an HTTP Connection but I’m unsure how to proceed.

If someone can point me in the right direction I would appreciate it!

This is the HTTP request I’ve been messing with but I’m not 100% sure it’s correct either:

https://bridge-online.cloud/COMPANYNAME-freshdesk/odata4

and then the table is: freshdesk.Freshdesk.tickets_incremental@fbt.

2

The following explanation might help you:

Otherwise, please schedule your organization’s free explanation session listed in the emails sent. The free sessions are also available with the Free Plan which Freshdesk is in.

3

Thank you, I’m having issues so I will schedule. I am now getting this message through Fabric:

Failed to create odata connection to RequestUrl.
Failed to get metadata of odata service, please check if service url and credential is correct and your application has permission to the resource.
Expected status code: 200, actual status code: 429, response is :

{"error":{"code":"itgenboe405","message":"The daily limit of 10,800,000 ms of fresh duration has been exceeded by 14,292,106 ms.","target":null,"details":[{"code":"TraceIdentifier","target":null,"message":"0HN76IIQH3436:00000001"},{"code":"Name","target":null,"message":"TotalDurationMsPerDay"},{"code":"LimitValue","target":null,"message":"10800000"},{"code":"CurrentValue","target":null,"message":"25092106"},{"code":"RemainingValue","target":null,"message":"0"},{"code":"RemainingDurationWindowSec","target":null,"message":"22886"},{"code":"LifetimeSecFromUtc","target":null,"message":"86400"},{"code":"SessionId","target":null,"message":"user-854aef17-d7b9-42ea-bbb8-4bfcaffea19b"},{"code":"DatabaseUid","target":null,"message":"92494436-d63d-432c-9dfa-28916b832096"},{"code":"SubscriptionType","target":null,"message":"SUBAATR"},{"code":"SubscriptionCode","target":null,"message":"L640337149"}],"innererror":null}}
4

The cause is that the tickets_incremental is downloaded fresh for over 10 times per day, as indicated by searching on a URL in Session I/Os containing ‘2020-11-25’. All returning 179.972 bytes of data with table name SyncTickets on a URL like:

https://acme.freshdesk.com/api/v2/tickets?per_page=100&page=1&updated_since=2022-11-25T00%3A00%3A00Z&order_by=updated_at&order_type=asc&include=stats

In fact, the data should be taken for the 2nd and next times from an incremental cache, and not re-loaded fresh from Freshdesk.

Using delegation, the problem will be investigated. Data volume is around 100.000 tickets on the Estate plan of Freshdesk.

5

Through a number of tests on delegation, an effort was made to reproduce the issue. The problem could not be reproduced. Typical download time was after initial load was 1 minute for approximately 100.000 tickets.

In case the problem persists, please add a reply to this one.