Powershell CSV to SQL Query

Question

I am back on my on-going learning into powershell, I have hit the limits of my capabilities. So wondering if anyone can throw me in the right direction and or tell me if what I am wanting to do is possible?

I have written a script which downloads a number of file(s) from the web and then deletes out all but the data that I wish to work with. These are .csv's...

I have added the Powershell code that I have managed to cobble together so far, when the data is downloaded see below. The data extract of one of these files looks like this.

Column A contains a description

Column B contains an ISIN (this is the working information)

Column C contains a numerical figure

Column D is where I want the SQL query data to be returned to

Please find a link to the sample data Data Example Data Example

What I then want to achieve is to run the following SQL command based on the data within Column B

Select *
From CL
Where CLISIN in ('GB0004835483',
'BE0003793107',
'GB00B7V2GY97',
'GB0000595859',
'GB00B1VCNQ84',
'GB0004992003',
'GB0002369352')

I believe that I will need to export this as another file perhaps? Though using the final results that are exported I then need to place these within Column D in this csv file.

Hoping that I have made this clear, if not please let me know and I will be as expansive as possible.

Long and the short is can I use powershell to automatically run a query for each item in column B and add the results of that matching query to the correlating line in column D?

I cannot find the answer via google nor here...

#### DOWNLOAD LOCATIONS ####
$DownloadPTMLocation =  "L:\Operations Database\TakeOverPanel\PTMDisclosureTable.xls"
$DownloadPTMCSVLocation =  "L:\Operations Database\TakeOverPanel\PTMDisclosureTable.csv"
$DownloadIPTMLocation = "L:\Operations Database\TakeOverPanel\IPTMDisclosureTable.xls"
$DownloadIPTMCSVLocation = "L:\Operations Database\TakeOverPanel\IPTMDisclosureTable.csv"

#### WEB URLS ###
$PTMURL = "http://www.thetakeoverpanel.org.uk/new/disclosureTable/v3/disclosuretable.xls"
$PTMCSVURL = "http://www.thetakeoverpanel.org.uk/new/disclosureTable/v3/disclosuretable.csv"
$IPTMURL = "http://irishtakeoverpanel.ie/disclosure/disclosuretable.xls"
$IPTMCSVURL = "http://irishtakeoverpanel.ie/disclosure/disclosuretable.csv"

$Path = Get-Location

#### Load Web Service #### 
$WebClient = New-Object System.Net.WebClient

#### Start download Process ####
Write-Host "Downloading PTM File 1 of 4" $Path -ForegroundColor Green
    $Url = $PTMURL
    $Path = $DownloadPTMLocation
    $WebClient.DownloadFile($PTMURL, $DownloadPTMLocation)

Write-Host "Downloading IPTM File 2 of 3" $Path -ForegroundColor Blue
    $Url = $IPTMURL 
    $Path = $DownloadIPTMLocation
    $WebClient.DownloadFile($IPTMURL, $DownloadIPTMLocation)

Write-Host "Downloading PTM Csv File 3 of 4" $Path -ForegroundColor Gray
    $Url = $PTMCSVURL
    $Path = $DownloadPTMCSVLocation
    $WebClient.DownloadFile($PTMCSVURL, $DownloadPTMCSVLocation)

Write-Host "Downloading IPTM File 4 of 4" $Path -ForegroundColor Red
    $Url = $IPTMCSVURL 
    $Path = $DownloadIPTMCSVLocation
    $WebClient.DownloadFile($IPTMCSVURL, $DownloadIPTMCSVLocation)



    #####################


    ## PTM ## 

#Customise Vars
$DownloadPTMCSVLocation = "L:\Operations Database\TakeOverPanel\PTMDisclosureTable.csv"
$OutputPTMCSVLocation = "L:\Operations Database\TakeOverPanel\PTMDisclosureTableb.csv"
$Match = "ISIN"
$Matchs = "NSI"

## Strips all lines that do not contain ISIN ##

(Get-Content $DownloadPTMCSVLocation) -match $Match | Out-File $OutputPTMCSVLocation

Remove-Item $DownloadPTMCSVLocation

Rename-Item $OutputPTMCSVLocation -NewName $DownloadPTMCSVLocation

(Get-Content $DownloadPTMCSVLocation) -match $Matchs | Out-File $OutputPTMCSVLocation

Remove-Item $DownloadPTMCSVLocation

Rename-Item $OutputPTMCSVLocation -NewName $DownloadPTMCSVLocation

Get-Content $DownloadPTMCSVLocation | % {
    $_ -replace 'ISIN: ',''
    } | % {
    $_ -replace 'NSI: ',''
    } | Set-Content $OutputPTMCSVLocation

So working on this further and having hit the wall (once more returning SQL query) here is the working code so far...

###########
## BEGIN ##
###########

#### DOWNLOAD LOCATIONS ####

$DownloadPTMCSVLocation =  "L:\Operations Database\TakeOverPanel\PTMDisclosureTable.csv"
$OutputPTMCSVLocation = "L:\Operations Database\TakeOverPanel\PTMDisclosureTableb.csv"


#### WEB URLS ###

$PTMCSVURL = "http://www.thetakeoverpanel.org.uk/new/disclosureTable/v3/disclosuretable.csv"


$Path = Get-Location

#### Load Web Service #### 
$WebClient = New-Object System.Net.WebClient

#### Start download Process ####


Write-Host "Downloading PTM Csv" $Path -ForegroundColor Gray
    $Url = $PTMCSVURL
    $Path = $DownloadPTMCSVLocation
    $WebClient.DownloadFile($PTMCSVURL, $DownloadPTMCSVLocation)



#####################


## PTM ## 

$Match = "ISIN"
$Matchs = "NSI"

Import-Csv $DownloadPTMCSVLocation -Header @("A", "ISIN", "NSI", "Output") | #Import the CSV
  Where { $_.ISIN -like "ISIN: ????????????" -and $_.NSI -like "NSI:*" } | #Filter rows
  Foreach-Object {
    $_.ISIN = $_.ISIN.Replace("ISIN: ", "")
    $_.NSI = $_.NSI.Replace("NSI: ", "")

    $query = "select CLIALPHASORTCODE, vl2securitynum, sum(cast(vl2beneficial as float)) as beneficial 
    from t5vaultsl2 vl2 left outer join t5client cli
    on vl2.VL2CLIENTNUM = cli.CLICODE

    where vl2.vl2securitynum = '$($_.ISIN)'

    group by CLIALPHASORTCODE, VL2SECURITYNUM"


##Credentials## 

$MISA = 'xx.xx.x.xx'
$MISB = 'xx.xx.x.xx'
$userName = 'UN'
$PassWord='PW'
$DB = 'reporting'


## CREATE MIS CREDENTIALS ##
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Data Source=$MISA;Initial Catalog=$DB;
Initial Catalog=$DB;User ID=$userName;Password=$PassWord;"

## - Runs Script from Set Location

$SqlCmd = New-Object System.Data.SqlClient.SqlCommand;
$SqlCMD.CommandText = $query;
$SqlCmd.Connection = $SqlConnection;

So script is now as follows...

#### DOWNLOAD LOCATIONS ####

$DownloadPTMCSVLocation =  "L:\Operations Database\TakeOverPanel\PTMDisclosureTable.csv"
$OutputPTMCSVLocation = "L:\Operations Database\TakeOverPanel\PTMDisclosureTableb.csv"
$ExportLocation = "L:\Operations Database\TakeOverPanel\test.csv"
$ExportLocationb = "L:\Operations Database\TakeOverPanel\test.xml"


#### WEB URLS ###

$PTMCSVURL = "http://www.thetakeoverpanel.org.uk/new/disclosureTable/v3/disclosuretable.csv"
$Path = Get-Location


#### Load Web Service #### 
$WebClient = New-Object System.Net.WebClient

#### Start download Process ####


Write-Host "Downloading PTM Csv" $Path -ForegroundColor Gray
    $Url = $PTMCSVURL
    $Path = $DownloadPTMCSVLocation
    $WebClient.DownloadFile($PTMCSVURL, $DownloadPTMCSVLocation)



################
#### Query  ####
################

$query = @"
    select CLIALPHASORTCODE, vl2securitynum, sum(cast(vl2beneficial as float)) as beneficial 
    from t5vaultsl2 vl2 left outer join t5client cli
    on vl2.VL2CLIENTNUM = cli.CLICODE

    where vl2.vl2securitynum = '$($_.ISIN)'

    group by CLIALPHASORTCODE, VL2SECURITYNUM
"@;


#####################################

$Match = "ISIN"
$Matchs = "NSI"

## Prepare SQL ##

$MISA = 'xx.xx.x.xx'
$MISB = 'xx.xx.x.xx'
$userName = 'UN'
$PassWord='PW'
$DB = 'reporting'


## CREATE SQL Connection ##
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Data Source=$MISA;Initial Catalog=$DB;Initial Catalog=$DB;User ID=$userName;Password=$PassWord;"
$SqlConnection.Open()

# Put everything in a Try block so if there is an error the SQL
# connection is still closed
try
{
    $SqlCmd = $SqlConnection.CreateCommand()

    ## Process CSV ##

    Import-Csv $DownloadPTMCSVLocation -Header @("A", "ISIN", "NSI", "Output") | #Import the CSV
      Where { $_.ISIN -like "ISIN: ????????????" -and $_.NSI -like "NSI:*" } | #Filter rows
      Foreach-Object {
        $_.ISIN = $_.ISIN.Replace("ISIN: ", "")
        $_.NSI = $_.NSI.Replace("NSI: ", "")

        # Get data from SQL
        $query = "select CLIALPHASORTCODE, vl2securitynum, sum(cast(vl2beneficial as float)) as beneficial from t5vaultsl2 vl2 left outer join t5client cli on vl2.VL2CLIENTNUM = cli.CLICODE where vl2.vl2securitynum = '$($_.ISIN)' group by CLIALPHASORTCODE, VL2SECURITYNUM"

        $SqlCmd.CommandText  = $query
        $result = $SqlCmd.ExecuteReader()
        $table = New-Object "System.Data.DataTable"
        $table.Load($result)

        $t = $table.Vl2Beneficial

        # Pass row on through the pipeline
        $_.Output = $table.VL2Beneficial

        $table | Export-Clixml $ExportLocationb

        Write-Output $_
    } | Export-Csv $OutputPTMCSVLocation -NoTypeInformation
}
finally
{
    # Always close SQL connection even if error is encountered.
    $SqlConnection.Close()
}

Please provide data from your CSV file (not screenshot). And what output you want to get from that CSV. If this CSV, a with determined structure you can use OPENROWSET to read data from this files on SQL Server side. — gofr1
– gofr1, Commented Aug 15, 2016 at 9:31
The PTM files can be downloaded from the provided links which should be enough — Deadly-Bagel
– Deadly-Bagel, Commented Aug 15, 2016 at 9:47
Hey Deadly_Bagel, Yes they can be downloaded and within my code I am doing so, what I am then doing is trying to compare my holdings against these PTM files. Which is where the SQL requirements come in. Gofr1, required changes have now been added I trust... The output that I wish to obtain within column D is my numerical position. — Mr Deans
– Mr Deans, Commented Aug 15, 2016 at 9:53
If I understand correctly, you need to generate a SQL query as per your example using column B of the spreadsheet you are generating, then put each respective result from said SQL query in column D? — Deadly-Bagel
– Deadly-Bagel, Commented Aug 15, 2016 at 9:55
Deadly-Bagel, exactly... I can think of other long winded ways of doing this but I really wish to contain everything within powershell if I can at all help it. — Mr Deans
– Mr Deans, Commented Aug 15, 2016 at 9:56

Deadly-Bagel · Accepted Answer · 2016-08-15 15:04:27Z

1

## PTM ## 

$Match = "ISIN"
$Matchs = "NSI"

## Prepare SQL ##

$MISA = 'xx.xx.x.xx'
$MISB = 'xx.xx.x.xx'
$userName = 'UN'
$PassWord='PW'
$DB = 'reporting'

## CREATE SQL Connection ##
$SqlConnection = New-Object System.Data.SqlClient.SqlConnection
$SqlConnection.ConnectionString = "Data Source=$MISA;Initial Catalog=$DB;Initial Catalog=$DB;User ID=$userName;Password=$PassWord;"
$SqlConnection.Open()

# Put everything in a Try block so if there is an error the SQL
# connection is still closed
try
{
    $SqlCmd = $SqlConnection.CreateCommand()

    ## Process CSV ##

    Import-Csv $DownloadPTMCSVLocation -Header @("A", "ISIN", "NSI", "Output") | #Import the CSV
      Where { $_.ISIN -like "ISIN: ????????????" -and $_.NSI -like "NSI:*" } | #Filter rows
      Foreach-Object {
        $_.ISIN = $_.ISIN.Replace("ISIN: ", "")
        $_.NSI = $_.NSI.Replace("NSI: ", "")

        # Get data from SQL
        $query = "select CLIALPHASORTCODE, vl2securitynum, sum(cast(vl2beneficial as float)) as beneficial from t5vaultsl2 vl2 left outer join t5client cli on vl2.VL2CLIENTNUM = cli.CLICODE where vl2.vl2securitynum = '$($_.ISIN)' group by CLIALPHASORTCODE, VL2SECURITYNUM"

        $SqlCmd.CommandText  = $query
        $result = $SqlCmd.ExecuteReader()
        $table = New-Object "System.Data.DataTable"
        $table.Load($result)

        # Pass row on through the pipeline
        $_.Output = $table.VL2Beneficial

        Write-Output $_
    } | Export-Csv $OutputPTMCSVLocation -NoTypeInformation
}
finally
{
    # Always close SQL connection even if error is encountered.
    $SqlConnection.Close()
}

Basically I threw out all the file renaming as it's completely unnecessary (use variables for that sort of thing if you need to), Import-Csv returns an array of objects, one for each row so I pipe them to the filter (where ISIN is checked to be 12 chars only) then loops through sorting out the data.

edited Aug 15, 2016 at 15:04

answered Aug 15, 2016 at 11:08

Deadly-Bagel

1,6201 gold badge10 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Mr Deans Over a year ago

Dropping you something back in chat... Further trying to get the SQL to output and back to my huge lack of knowledge frustrating me... I can either get it to return nothing in column D, or the SQL query but not the results of the sql query...

Mr Deans Over a year ago

Final element, looks to be an error on export with System.Object[] being returned into the results. Any thoughts... I can send a full update on code if required?

Deadly-Bagel Over a year ago

No I think I know what's happening. See chat.

Collectives™ on Stack Overflow

Powershell CSV to SQL Query

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest