Programming & Data Science

 

 

 Data Science Resource

 

Blogs

http://flowingdata.com/2012/04/27/data-and-visualization-blogs-worth-following/

https://towardsdatascience.com

http://www.mightywebdeveloper.com/

 

Python - Convert CSV to JSON

https://www.idiotinside.com/2015/09/18/csv-json-pretty-print-python/

 

dBase Web App

http://www.dbase.com/Knowledgebase/INT/WebAppHowTos/CREATINGADB2KWEBAPP.HTM

 

Connect to a dBase or Other DBF file

https://docs.microsoft.com/en-us/sql/integration-services/connection-manager/connect-to-a-dbase-or-other-dbf-file

 

Data Converters

http://okfnlabs.org/dataconverters/

 

Tutorials

https://schoolofdata.org/courses/sql-databases-part-two/

http://www.graphics.stanford.edu/courses/cs448b-04-winter/

- Python

https://www.cybrhome.com/topic/python-tutorials

Javascript

https://javascript.info/intro

 Powershell

https://soykablog.wordpress.com

Visualisation Tools and Resources

https://drawingbynumbers.org/toolsandresources

Drawing by numbers organizes data visualization tools under four categories - Charts and graphs; Mapping and mashups; Design, layout and presentation; and Data management tools. Browse through them below.

However advanced their capabilities, most of the tools profiled here address fairly simple data and design tasks. They also have feature sets that anyone with a little technical knowledge (and some persistence) could use to make something. They all have acceptable support structures and materials and a degree of non-English language support. They are also all free to use or open source. There are commercial alternatives to these tools as well, and we will be profiling them soon.

 

D3.js Tutorials

https://github.com/uwdata/d3-tutorials

D3 Tutorials for CSE512 Data Visualization Course at the University of Washington by @kanitw and @domoritz at @uwdata.

Our wiki also lists a number of useful resources for visualization design and development. Feel free to use them for your classes or workshops and send us feedback and improvements. If you find these tutorial useful, please tweet to us at @uwdata and write in our Testimonials page.

http://chimera.labs.oreilly.com/books/1230000000345/index.html

 

Javascript

https://developer.mozilla.org/en-US/docs/Web/JavaScript

 http://bonsaiden.github.io/JavaScript-Garden/

 https://wesbos.com/blog/

Reading CSV with JS and HTML5

https://mounirmesselmeni.github.io/2012/11/20/reading-csv-file-with-javascript-and-html5-file-api/

https://coderexample.com/reading-csv-file-using-javascript/ 

https://code.tutsplus.com/tutorials/parsing-a-csv-file-with-javascript--cms-25626

https://www.papaparse.com/#local-files

Developers to Follow:

https://theelous3.net/index

Javascript Tutorials on Github:

https://github.com/freeCodeCamp/freeCodeCamp

https://github.com/getify/You-Dont-Know-JS

https://github.com/trekhleb/javascript-algorithms

https://github.com/30-seconds/30-seconds-of-code

https://github.com/leonardomso/33-js-concepts

 

Node

CSV to JSON

https://github.com/Keyang/node-csvtojson 

http://keyangxiang.com/csvtojson/

Guides

Regex

http://www.zytrax.com/tech/web/regex.htm

https://www.cheatography.com/davechild/cheat-sheets/regular-expressions/

http://www.regexbuddy.com/

* Ruby Regex expression editor

http://www.rubular.com/ 

 

JQuery Themes

http://jqueryui.com/themeroller/

https://www.ibm.com/developerworks/library/wa-jquerythemes/

 

Prepare to Sign Code by Installing the Windows SDK

In order to use SignTool.exe to sign your application, you need to either install Microsoft Visual Studio 2005 (or later), or on the machine where you will be signing code, download and install one of the following versions of Microsoft Windows SDK:

You can sign software automatically upon building in Visual Studio by adding the following command to the project as a "Post-build event command line" (project properties > Build Events):

"C:\PATH-TO-WINDOWS-SDK\signtool.exe" sign /fd SHA256 /f "C:\(ProjectDir)SigningCert.pfx" /p Password /t http://timestamp.digicert.com

Source: 

https://docs.microsoft.com/en-us/windows/desktop/appxpkg/how-to-sign-a-package-using-signtool

https://www.digicert.com/code-signing/signcode-signtool-command-line.htm

http://msdn.microsoft.com/en-us/library/ms180786%28VS.80%29.aspx

Basics of Signing and Verifying code - CodeProjecthttps://www.codeproject.com › General Programming › Cryptography & Security

 

PHP PROGRAMMING

PHP allow access based on IP addresses:

Example, using true or false statements.

<?php

$custome_name = array('10.10.10.10, 20.20.20.20)';

$custome_name2 = false;

if (in_array($_SERVER['REMOTE_ADDR'], $custome_name)){

     $custome_name2 = true;

?>

This returns a value if the IP address is not present: <?php if (!$custome_name2) { ?>

More about IF statements can be found here: http://www.zentut.com/php-tutorial/php-if/

If you were to use $_SERVER['HTTP_HOST'], this would return a value in HTTP header and is easily faked. Using remote_addr the result returns the address to Apache. Here is another example from here: https://daveismyname.blog/grant-access-based-on-ip-address-with-php

//collection of allowed IP addresses
$allowlist = array(
    '36.285.23.23',
    '12.101.67.56',
    '98.465.23.89',
    '16.289.90.10',
    '71.214.228.18'
);

//if users IP is not in allowed list kill the script
if(!in_array($_SERVER['REMOTE_ADDR'],$allowlist)){
    die('This website cannot be accessed from your location.');
}
//if users IP is not in allowed list kill the script
if(!in_array($_SERVER['REMOTE_ADDR'],$allowlist)){
    die('This website cannot be accessed from your location.');
}

 

What does $("#value"); mean?

This is known as a jQuery Selector. It can be used to get certain elements from a website. See below:

#id $("#lastname") The element with id="lastname"
.class $(".intro") All elements with class="intro"

https://www.w3schools.com/jquery/jquery_ref_selectors.asp

 

 

YouTube Downloader:

https://blog.usejournal.com/how-i-made-my-own-youtube-downloader-using-javascript-and-node-js-160b172f6e10

Index.html:

<!DOCTYPE html>
<html>
<head>
<title>YouTube Downloader</title>
</head>
<body>
<h1 class="heading">My Own YouTube Downloader !</h1>
<input class="URL-input" placeholder="Video URL e.g. https://www.youtube.com/watch?v=MtN1YnoL46Q">
<button class="convert-button">Convert</button>
</body>
</html>

 

CSS

After that we will add some styling to our page using CSS. You can use CSS frameworks like Bootstrap or Bulma. So first I linked my CSS file (style.css) by adding the link tag inside the head:

<link rel="stylesheet" href="style.css">

Style.css:

* {
text-align: center;
}
.heading {
font-family: Arial;
margin-top:40vh;
}
.URL-input, .convert-button {
font-size:1.3em;
padding:5px 10px;
}
.URL-input {
border-radius:4px 0px 0px 4px;
width:30em;
text-align: left;
border:2px solid #EEEEEE;
background: #EEEEEE;
outline:none;
}
.URL-input:focus {
border:2px solid #0485ff;
}
.convert-button {
border-radius:0px 4px 4px 0px;
border:2px solid #0485ff;
background: #0485ff;
color:white;
}

 

JavaScript and Node.js

Then now we need to setup our JavaScript and our Node.js Server. First let’s create a JavaScript file. Let’s call it script.js. Then we are going to link it to our HTML File by adding script tags in the bottom (Very Important)

<script src='script.js'></script>

In out script.js file we are going to write that little piece of code output to the console the value of the input. We are doing that to check that everything is working.

var convertBtn = document.querySelector('.convert-button');
var URLinput = document.querySelector('.URL-input');
convertBtn.addEventListener('click', () => {
console.log(`URL: ${URLinput.value}`);
sendURL(URLinput.value);
});
function sendURL(URL) {
// We will put code here later
}

//We will put code here later: window.location.href = `http://localhost:4000/download?URL=${URL}`;

The function sendURL() will send the URL to our server. Now we are going to start writing code for our Server using Node.js. First, we are going to create a folder called Server then create a file called index.js inside the Server folder. This will have the code for the server to run. After creating the file, we need to run this command in the command line or terminal in the Server folder:

npm init

You will be asked a series of questions about the project. All the default responses would be fine. After that we are going to install some packages that we are going to use.

Type the command below to install the required packages which are express, ytdl-core and cors

npm install express cors ytdl-core

If you get the following errors:

>npm install express cors ytdl-core
npm ERR! code SELF_SIGNED_CERT_IN_CHAIN
npm ERR! errno SELF_SIGNED_CERT_IN_CHAIN
npm ERR! request to https://registry.npmjs.org/cors failed, reason: self signed certificate in certificate chain

You will need to run the following command. The reason behind this is due to a firewall or proxy on your network requiring SSL verification. 

npm config set strict-ssl false

Source:https://stackoverflow.com/questions/9626990/receiving-error-error-ssl-error-self-signed-cert-in-chain-while-using-npm

The function sendURL() will send the URL to our server. Now we are going to start writing code for our Server using Node.js. First, we are going to create a folder called Server then create a file called index.js inside the Server folder. This will have the code for the server to run. After creating the file, we need to run this command in the command line or terminal in the Server folder:

npm init

You will be asked a series of questions about the project. All the default responses would be fine. After that we are going to install some packages that we are going to use.

Type the command below to install the required packages which are express, ytdl-core and cors

npm install express cors ytdl-core

To setup our server we need to type some code inside the index.js file. We need to require the packages and setup the server:

const express = require('express');
const cors = require('cors');
const ytdl = require('ytdl-core');
const app = express();
app.listen(4000, () => {
console.log('Server Works !!! At port 4000');
});

Here is a little Setup. We required the packages and made the server listen on port 4000.

If we type in the command line/terminal:

node index.js

we should see Server Works !!! At port 4000 . If you didn’t get that you have done something wrong.

After that we need to make the server listen for get request on the path /download . So, we added this:

const express = require('express');
const cors = require('cors');
const ytdl = require('ytdl-core');
const app = express();
app.use(cors());
app.listen(4000, () => {
console.log('Server Works !!! At port 4000');
});
app.get('/download', (req,res) => {
var URL = req.query.URL;
res.json({url:URL});
})

so, when the user sends a GET request to the server at the path /downloadthen the server will respond with the query inserted with the request. Let’s go back to the front-end and try to make JavaScript send a GET request to the server. Also, I used the middleware cors() so my server can receive requests. I used fetch() to send the request from JavaScript:

var convertBtn = document.querySelector('.convert-button');
var URLinput = document.querySelector('.URL-input');
convertBtn.addEventListener('click', () => {
console.log(`URL: ${URLinput.value}`);
sendURL(URLinput.value);
});
function sendURL(URL) {
fetch(`http://localhost:4000/download?URL=${URL}`, {
method:'GET'
}).then(res => res.json())
.then(json => console.log(json));
}

So, we can see if we click on the button, we get a response back from the server with the URL that we send. Now we need to use ytdl-core which stand for YouTube DownLoader core which will download the video for us. Let’s go back to index.js. We are going to make it download the video then send it:

const express = require('express');
const cors = require('cors');
const ytdl = require('ytdl-core');
const app = express();
app.use(cors());
app.listen(4000, () => {
console.log('Server Works !!! At port 4000');
});
app.get('/download', (req,res) => {
var URL = req.query.URL;
res.header('Content-Disposition', 'attachment; filename="video.mp4"');
ytdl(URL, {
format: 'mp4'
}).pipe(res);
});

Basically, what is it doing is that when the user send a GET request it take the query given in the URL then adds a header that says any attachment sent needs to be called video.mp4. then use ytdl to download the video with format mp4 then pipe it (send it) to the user to download.

When you run this and test it, you will quickly find out that it doesn’t work. That is because the request needs to be in the URL rather than as a fetch. So, we can basically replace the fetch with a redirect to the URL to download the video.

var convertBtn = document.querySelector('.convert-button');
var URLinput = document.querySelector('.URL-input');
convertBtn.addEventListener('click', () => {
console.log(`URL: ${URLinput.value}`);
sendURL(URLinput.value);
});
function sendURL(URL) {
window.location.href = `http://localhost:4000/download?URL=${URL}`;
}

and Now you will have your very own YouTube Downloader. You can also play around with the ytdl-core and try to add .mp3 or .flv formats. You can also do what I done before which is linking the YouTube Data API with your web application. Possibly Searching for a video then downloading it.

ytdl-core

ytdl-core is a module that can be installed in your node project. It allows you to download youtube videos. This awsome module was made by a person called fent. It has a lot of features which are great like ytdl.videoInfo which gives you different information about a video. I have use this feature to create a youtube downloader where you can search for the video or put the link in to download the video. ytdl-core can be used for all different purposes. Let me give you an example. If you want your website to have your video that you uploaded to youtube and you also don’t want it to be blocked in places where youtube is blocked like Schools. Then you can use ytdl-core to download your own video and use it. ytdl also allows you to validate Youtube URLs and validate Youtube video IDs

IF you see the following error in cmd: 

Error: self signed certificate in certificate chain
at TLSSocket.onConnectSecure (_tls_wrap.js:1177:34)
at TLSSocket.emit (events.js:197:13)
at TLSSocket._finishInit (_tls_wrap.js:668:8)
Emitted 'error' event at:
at ytdl.getInfo (X:\Desktop\Youtube-dl\Server\node_modules\ytdl-core\lib\index.js:19:14)
at fn (X:\Desktop\Youtube-dl\Server\node_modules\ytdl-core\lib\info.js:362:25)
at exports.getBasicInfo (X:\Desktop\Youtube-dl\Server\node_modules\ytdl-core\lib\info.js:189:21)
at fn (X:\Desktop\Youtube-dl\Server\node_modules\ytdl-core\lib\info.js:362:25)
at PassThrough.request (X:\Desktop\Youtube-dl\Server\node_modules\ytdl-core\lib\info.js:44:21)
at PassThrough.emit (events.js:197:13)
at ClientRequest.onRequestError (X:\Desktop\Youtube-dl\Server\node_modules\miniget\lib\index.js:83:12)
at ClientRequest.emit (events.js:197:13)
at TLSSocket.socketErrorListener (_http_client.js:397:9)
at TLSSocket.emit (events.js:197:13)

QUICK FIX: DANGEROUS This disables HTTPS / SSL / TLS checking across your entire node.js environment. Please see the solution using an https agent below.

You will need to add:

process.env["NODE_TLS_REJECT_UNAUTHORIZED"] = 0;

In code, before calling app. get

Source: https://stackoverflow.com/questions/20433287/node-js-request-cert-has-expired#answer-29397100

Source: https://stackoverflow.com/questions/10888610/ignore-invalid-self-signed-ssl-certificate-in-node-js-with-https-request

This article was taken from here: https://blog.usejournal.com/how-i-made-my-own-youtube-downloader-using-javascript-and-node-js-160b172f6e10

 

 

Active Directory user filter does not search nested groups (PHP)

What is the string 1.2.840.113556.1.4.1941???

Problem

An Active Directory (AD) user object filter to pull in users from a specific group does not recursively search groups nested under the specified group, even though recursion is enabled. A filter like the following is used:

(&(objectCategory=Person)(sAMAccountName=*)(memberOf=cn=TestGroup,ou=Groups,ou=CompanyUsers,dc=test,dc=corp))

Cause

Crowd uses basic LDAP syntax rules for searching. By default, any searches with memberOf will only check direct attributes, so AD will only return information back to Crowd based on direct attribute checks.

To get a recursive search, or to have AD check relations, extra properties need to be included to the filter. In this case, the string 1.2.840.113556.1.4.1941 will need to be added. According to Microsoft:

The string 1.2.840.113556.1.4.1941 specifies LDAP_MATCHING_RULE_IN_CHAIN. This applies only to DN attributes. This is an extended match operator that walks the chain of ancestry in objects all the way to the root until it finds a match. This reveals group nesting. It is available only on domain controllers with Windows Server 2003 SP2 or Windows Server 2008 (or above).

For more information, see the following from Technet:

Resolution 

  • Modify the above filter to include the extended match operator:

    (&(objectCategory=Person)(sAMAccountName=*)(memberOf:1.2.840.113556.1.4.1941:=cn=TestGroup,ou=Groups,ou=CompanyUsers,dc=test,dc=corp))
    

 

Last modified on Oct 16, 2018
 
-->
© Donald Young HTML Responcive Template designed by Crazy Pixels