chinwag/chinwagsocial
chinwag/chinwagsocial
2
0
Fork 0
Code Issues Pull requests Releases 14 Wiki Activity

Change domain block CSV parsing to be more robust and handle more lists (#21470)

Browse source 
* Change domain block CSV parsing to be more robust and handle more lists

* Add some tests

* Improve domain block import validation and reporting
This commit is contained in:
Claire 2023-01-18 16:20:52 +01:00 • committed by GitHub
commit fcc4c9b34a
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
8 changed files with 90 additions and 39 deletions
Download patch file Download diff file Expand all files Collapse all files

47
app/models/admin/import.rb
View file

@ -1,5 +1,7 @@
# frozen_string_literal: true
require 'csv'
# A non-activerecord helper class for csv upload
class Admin::Import
include ActiveModel::Model
@ -15,17 +17,46 @@ class Admin::Import
data.original_filename
end
def csv_rows
csv_data.rewind
csv_data.take(ROWS_PROCESSING_LIMIT + 1)
end
private
def csv_data
return @csv_data if defined?(@csv_data)
csv_converter = lambda do |field, field_info|
case field_info.header
when '#domain', '#public_comment'
field&.strip
when '#severity'
field&.strip&.to_sym
when '#reject_media', '#reject_reports', '#obfuscate'
ActiveModel::Type::Boolean.new.cast(field)
else
field
end
end
@csv_data = CSV.open(data.path, encoding: 'UTF-8', skip_blanks: true, headers: true, converters: csv_converter)
@csv_data.take(1) # Ensure the headers are read
@csv_data = CSV.open(data.path, encoding: 'UTF-8', skip_blanks: true, headers: ['#domain'], converters: csv_converter) unless @csv_data.headers&.first == '#domain'
@csv_data
end
def csv_row_count
return @csv_row_count if defined?(@csv_row_count)
csv_data.rewind
@csv_row_count = csv_data.take(ROWS_PROCESSING_LIMIT + 2).count
end
def validate_data
return if data.blank?
csv_data = CSV.read(data.path, encoding: 'UTF-8')
row_count = csv_data.size
row_count -= 1 if csv_data.first&.first == '#domain'
errors.add(:data, I18n.t('imports.errors.over_rows_processing_limit', count: ROWS_PROCESSING_LIMIT)) if row_count > ROWS_PROCESSING_LIMIT
return if data.nil?
errors.add(:data, I18n.t('imports.errors.over_rows_processing_limit', count: ROWS_PROCESSING_LIMIT)) if csv_row_count > ROWS_PROCESSING_LIMIT
rescue CSV::MalformedCSVError => e
errors.add(:data, I18n.t('imports.errors.invalid_csv_file', error: e.message))
end