• Agile Project Planning: A Step-by-Step Guide

    Image Source Guest Post by Gilad David Maayan Software teams have been adopting agile project management methodologies for nearly a decade. As a result, development teams have increased their velocity and communication, and they are quicker to react to market trends. However, to be truly effective, agile methodologies rely on solid planning. This article offers a guideline for planning your agile project. What Is Agile Project Planning? Agile project planning is a project management approach that breaks big projects into independent units called sprints. Each sprint is a short work cycle that runs from one week to one month.  Teams use sprints as small manageable tasks, which they strive to…

  • Create your first ETL in Luigi
    An introductory tutorial covering the basics of Luigi and an example ETL application.

    This post is the part of Data Engineering Series. In previous posts, I discussed writing ETLs in Bonobo, Spark, and Airflow. In this post, I am introducing another ETL tool which was developed by Spotify, called Luigi. Earlier I had discussed here, here and here about writing basic ETL pipelines. Bonobo is cool for write ETL pipelines but the world is not all about writing ETL pipelines to automate things. There are other use cases in which you have to perform tasks in a certain order once or periodically. For instance: Monitoring Cron jobs transferring data from one place to another. Automating your DevOps operations. Periodically fetching data from websites and…

  • Promote yourself on my blog

    I just realized that this bog is up for more than five years. I was not so active in the first two years hence this blog did not generate much traffic. In 2016 I resumed and then I have been writing since then. Oh by the way this blog is not my first attempt though as I have written  in past as well.  So far I have written 99% of these posts(quite obvious, right?) but I am the person who loves to share the knowledge I have and so far I am successful. Today I am (re)opening this platform for everyone who wants to share his/her creativity. This is not…

  • Securing Your Data in Azure: Tips and Tricks

    Guest Post by Gilad David Maayan Two-thirds of all businesses are currently using some form of cloud storage, with many turning to Azure as their provider. Cloud services contain a significant amount of data and allow persistent Internet access. This makes clouds an appealing and valuable target for attackers.  To make sure that your data doesn’t fall victim to attacks, you should take steps to protect your cloud systems. The first step is learning how your data is vulnerable and what steps you can take to secure it. In this article, you’ll learn about security concerns specific to Azure and some best practices for addressing these vulnerabilities. Azure Security Considerations…

  • ScrapeGen – Tool for generating Python scrapers
    A simple python tool that generates a requests/bs4 based web scraper

    OK, so I was kind of bored last week so thought of coming up something anyway, even if it is useless. So, I allocated a few hours and came up with ScrapeGen. What is ScrapeGen? It is a simple Python-based command-line tool that generates python web scrapers based on rules and details entered in a YAML file. When it runs, it generates a new file. Rules turned to separate functions which then are called to main parsing method. View the Demo: Why is needed? Such kind of tool could be good for companies and individuals who write many parsers and hardcode the rules within the .py files. Imagine the rule…

  • You don’t need to know everything

    Image Source Last week I decided to shut all the streams of political news and information. The purpose was to take a hiatus from all kind of information related to politics or world affairs. I wanted to confirm that knowing everything happening around me is not necessary and I am not the center of this universe, thus, if I do not know something, it will not stop working. This is not the first time I was doing, in the past, I have taken such breaks or I say, semi-breaks. This time it was different in a way that I was not only blocking all info from my laptop but also…

  • 5 Must-Know Web Application Security Tips

    Image Source Guest Post by Gilad David Maayan Gone are the days when developers could code a web application, release it, and be done with the project. Agile methodologies are changing the way we work, the way we code, and the way we collaborate. The developers of 2019 are expected to ensure the web apps they deliver are secure. That means your work doesn’t end with quick delivery. You also need to secure the apps. Read on to learn how to do that. 1. Don’t Aim to Fix All Vulnerabilities: Prioritize Them Common website platforms like WordPress, Drupal and Joomla are plagued with thousands of vulnerabilities. If you develop your…

  • Scraping dynamic websites using Scraper API and Python
    Learn how to efficiently and easily scrape modern Javascript enabled websites or Single Page Applications without installing a headless browser and Selenium

    In the last post of scraping series, I showed you how you can use Scraper API to scrape websites that use proxies hence your chance of getting blocked is reduced. Today I am going to show how you can use Scraper API to scrape websites that are using AJAX to render data with the help of JavaScript, Single Page Applications(SPAs) or scraping websites using frameworks like ReactJS, AngularJS or VueJS. I will be working on the same code I had written in the introductory post. Let’s work on a simple example. There is a website that tells your IP, called HttpBin. If you load via browser it will tell your…

  • Things every developer should know to improve site performance

    Introduction Hundreds of websites/web apps are being launched every day for different purposes. Poorly optimized websites not only leave a bad impact on visitors but also hurt businesses. While more and more people visiting websites on their mobile devices, it is very important to make them load fast. In this post, I am going to discuss a few things that every developer should know while building or updating a website to load them fast. I will categorize these issues into two sections: Frontend and Backend. Types of Optimization Frontend:- In which CSS, JS and at times HTML is refactored and optimized one way or other. Backend:- Databases; indices, queries etc.…

  • 5 Ways EBS Snapshots Can Make Your Life Easier

    Image source Guest Post by Gilad David Maayan Amazon Web Services is the leading cloud provider, offering a range of storage options to meet the complex and evolving needs of an organization. Along with the standard cloud storage option, S3, Amazon also offers EBS, which provides flexible storage for the data contained in applications. With Amazon’s high availability, customizability and variety of integrated cloud services, you can pursue a storage tiering strategy for your data, taking advantage of a key facet of Amazon—EBS snapshots. Read on to learn how snapshots can make your life on the cloud easier. What Is Amazon EBS? Amazon Elastic Block Store (EBS) is a cloud-based…